sqlite update in a loop take to long - vb.net
I am updating only 20 rows in a loop but the best performance I get is around 2.5 seconds for the 20 rows.
What can I do to make this simple update faster?
This is what Iv'e tried:
Dim connectionString As New SQLite.SQLiteConnection("data source=" & Application.StartupPath & "\db.db3; Version=3;")
Dim connection = New SQLite.SQLiteConnection(connectionString)
connection.Open()
Dim com3 As New System.Data.SQLite.SQLiteCommand(connection)
For Each row As DataRow In table.Rows
com3.CommandText = "UPDATE Results SET LastScore=" & row.Item("position") & " WHERE TeamID='" & row.Item("teamid") & "'"
com3.ExecuteNonQuery()
Next
connection.Close()
AND
Dim connectionString As New SQLite.SQLiteConnection("data source=" & Application.StartupPath & "\db.db3; Version=3;")
Using con As New SQLiteConnection(connectionString)
con.Open()
Using c As New SQLiteCommand(con)
c.CommandText = "UPDATE Results SET LastScore= #Pos WHERE TeamID = #ID"
For Each row As DataRow In table.Rows
c.Parameters.AddWithValue("#Pos", row.Item("position"))
c.Parameters.AddWithValue("#ID", row.Item("teamid"))
c.ExecuteNonQuery()
Next
End Using
con.Close()
End Using
Thanks to muffi's answer this is now resolved. Add BeginTransaction()
Dim connectionString As New SQLite.SQLiteConnection("data source=" & Application.StartupPath & "\db.db3; Version=3;")
Using con As New SQLiteConnection(connectionString)
con.Open()
Using t As SQLiteTransaction = con.BeginTransaction()
Using c As New SQLiteCommand(con)
c.CommandText = "UPDATE Results SET LastScore= #Pos WHERE TeamID = #ID"
For Each row As DataRow In table.Rows
c.Parameters.AddWithValue("#Pos", row.Item("position"))
c.Parameters.AddWithValue("#ID", row.Item("teamid"))
c.ExecuteNonQuery()
Next
End Using
t.Commit()
End Using
con.Close()
End Using
Related
select multiple column from excel in VBA
How can I select multiple column in Excel file using Vba. because I have excel file contains 580 columns. I try using Query below, but its only can select 255 columns. Dim ExcelConnection As New System.Data.OleDb.OleDbConnection("Provider=Microsoft.ACE.OLEDB.12.0; Data Source=" & TxtPath.Text & "; Extended Properties=""Excel 12.0 Xml; HDR=Yes""") ExcelConnection.Open() Dim expr As String = "select * FROM [Sheet1$]" Dim objCmdSelect As OleDbCommand = New OleDbCommand(expr, ExcelConnection) Dim objDR As OleDbDataReader Dim SQLconn As New SqlConnection() Dim ConnString As String = "data source=" & DBServer1 & "; initial catalog=" & DBDataBase1 & "; persist security info=true; user id=" & DBUSer1 & "; password=" & DBPass1 SQLconn.ConnectionString = ConnString SQLconn.Open() Using bulkCopy As SqlBulkCopy = New SqlBulkCopy(SQLconn) bulkCopy.DestinationTableName = "TMP_UPLOAD_ISP" Try objDR = objCmdSelect.ExecuteReader bulkCopy.WriteToServer(objDR) objDR.Close() SQLconn.Close() Catch ex As Exception MsgBox(ex.ToString) End Try I try by using select column header (220 Columns to select) ,its getting error until I reduce down to 100 columns. Dim ExcelConnection As New System.Data.OleDb.OleDbConnection("Provider=Microsoft.ACE.OLEDB.12.0; Data Source=" & TxtPath.Text & "; Extended Properties=""Excel 12.0 Xml; HDR=Yes""") ExcelConnection.Open() Dim expr As String = "select Plan_Year,Colom4,Colom6,Colom7,Colom21,Colom22,Colom24,Colom25,Colom27,Colom28,Colom33,Colom34,Colom38,Colom39,Colom41,Colom42,Colom47,Colom48,Colom49,Colom50,Colom51,Colom52,Colom66,Colom67,Colom69,Colom70,Colom72,Colom73,Colom78,Colom79,Colom83,Colom84,Colom86,Colom87,Colom92,Colom93,Colom94,Colom95,Colom96,Colom97,Colom111,Colom112,Colom114,Colom115,Colom117,Colom118,Colom123,Colom124,Colom128,Colom129,Colom131,Colom132,Colom137,Colom138,Colom139,Colom140,Colom141,Colom142,Colom156,Colom157,Colom159,Colom160,Colom162,Colom163,Colom168,Colom169,Colom173,Colom174,Colom176,Colom177,Colom182,Colom183,Colom184,Colom185,Colom186,Colom187,Colom201,Colom202,Colom204,Colom205,Colom207,Colom208,Colom213,Colom214,Colom218,Colom219,Colom221,Colom222,Colom227,Colom228,Colom229,Colom230,Colom231,Colom232,Colom246,Colom247,Colom249,Colom250,Colom252,Colom253,Colom258,Colom259,Colom263,Colom264,Colom266,Colom267,Colom272,Colom273,Colom274,Colom275,Colom276,Colom277,Colom303,Colom304,Colom306,Colom307,Colom309,Colom310,Colom315,Colom316,Colom320,Colom321,Colom323,Colom324,Colom329,Colom330,Colom331,Colom332,Colom333,Colom334,Colom348,Colom349,Colom351,Colom352,Colom354,Colom355,Colom360,Colom361,Colom365,Colom366,Colom368,Colom369,Colom374,Colom375,Colom376,Colom377,Colom378,Colom379,Colom393,Colom394,Colom396,Colom397,Colom399,Colom400,Colom406,Colom407,Colom410,Colom411,Colom413,Colom414,Colom419,Colom420,Colom421,Colom422,Colom423,Colom424,Colom438,Colom439,Colom441,Colom442,Colom444,Colom445,Colom450,Colom451,Colom455,Colom456,Colom458,Colom459,Colom464,Colom465,Colom466,Colom467,Colom468,Colom469,Colom483,Colom484,Colom486,Colom487,Colom489,Colom490,Colom495,Colom496,Colom500,Colom501,Colom503,Colom504,Colom509,Colom510,Colom511,Colom512,Colom513,Colom514,Colom528,Colom529,Colom531,Colom532,Colom534,Colom535,Colom540,Colom541,Colom545,Colom546,Colom548,Colom549,Colom554,Colom555,Colom556,Colom557,Colom558,Colom559 FROM [Sheet1$]" Dim objCmdSelect As OleDbCommand = New OleDbCommand(expr, ExcelConnection) Dim objDR As OleDbDataReader Dim SQLconn As New SqlConnection() Dim ConnString As String = "data source=" & DBServer1 & "; initial catalog=" & DBDataBase1 & "; persist security info=true; user id=" & DBUSer1 & "; password=" & DBPass1 SQLconn.ConnectionString = ConnString SQLconn.Open() Using bulkCopy As SqlBulkCopy = New SqlBulkCopy(SQLconn) bulkCopy.DestinationTableName = "TMP_UPLOAD_ISP" Try objDR = objCmdSelect.ExecuteReader bulkCopy.WriteToServer(objDR) objDR.Close() SQLconn.Close() Catch ex As Exception MsgBox(ex.ToString) End Try How can I Select column that I want to select (220 columns intermittently).
You are using the ACE provider and that means you are bound by the limitations of MS Access. Access has a maximum field limit of 255 (see: Access 2016 specifications). Depending on the source worksheet, you could use a range query to retrieve multiple result sets and then merge them. Select * From [SheetName$A1:IU] then Select * From [SheetName$UI1:SQ] Column "UI" is column number 255 and column "SQ" is column number 511. If you are not familiar with this range syntax, omitting the row on the end column tells it to seek from the starting row until no data is found in the column range.
vb.net Using SqlConnection doesn't free the file from use
I am working with a local .mdf file and I execute some queries to the database and I am using USING blocks to make sure the SqlConnection and SqlReader are disposed of correctly. I then try to read the data of the file to generate a MD5 Hash of the file but it says the file is still in use. The code isn't the cleanest it is my first time working with the Sql in a VB.NET app. SQL Insert: Dim finalW As String = "" Dim finalO() As String Dim currentcounter As Integer = 0 For Each Dir As String In System.IO.Directory.GetDirectories(Pathfinder) Dim dirInfo As New System.IO.DirectoryInfo(Dir) Dim temp As New List(Of String) For Each currentFile In Directory.GetFiles(Pathfinder & "\" & dirInfo.Name & "\", "*.png", SearchOption.TopDirectoryOnly) temp.Add(Path.GetFileName(currentFile)) Next If temp.Count <> 0 Then finalW = temp.Find(AddressOf GetNewIcon) finalO = temp.FindAll(AddressOf GetOldIcon).ToArray If finalW <> "" Then Using con As New SqlClient.SqlConnection("Data Source=(LocalDB)\MSSQLLocalDB;AttachDbFilename=""" & PathFinal & "ImaginiDB.mdf"";Integrated Security=True") con.Open() Using cmd = con.CreateCommand() cmd.CommandType = CommandType.Text cmd.CommandText = "INSERT NewIcon (Name) VALUES ('" & finalW.Trim() & "')" cmd.Connection = con cmd.ExecuteNonQuery() End Using currentcounter = currentcounter + 1 Dim Id As String = "" Using command = con.CreateCommand() command.CommandType = CommandType.Text command.CommandText = "SELECT * FROM NewIcon WHERE Name='" & finalW & "'" Using reader As SqlDataReader = command.ExecuteReader() While reader.Read() Id = reader(0) End While End Using End Using For Each item As String In finalO Using cmd2 = con.CreateCommand() cmd2.CommandType = CommandType.Text cmd2.CommandText = "INSERT OldIcon (NID,Name) VALUES ('" & Id & "','" & item.ToString.Trim() & "')" cmd2.Connection = con cmd2.ExecuteNonQuery() End Using currentcounter = currentcounter + 1 Next Dim cur As Long = currentcounter * 100 / counter SetProgress(cur) End Using End If End If Next GC.Collect() GC.WaitForPendingFinalizers() SetLabel4Text("FINISHED IMPORT", Color.Red) MD5 Generation ran after this process is finished: Public Function GenMD5(ByVal Filename As String) As String Dim MD5 = System.Security.Cryptography.MD5.Create Dim Hash As Byte() Dim sb As New System.Text.StringBuilder Using st As New IO.FileStream(Filename, IO.FileMode.Open, IO.FileAccess.Read, IO.FileShare.Read) Hash = MD5.ComputeHash(st) End Using For Each b In Hash sb.Append(b.ToString("X2")) Next Return sb.ToString End Function
So as mentioned in the comment by #jmcilhinney Different connections use different pool as the MSDN says: When a connection is first opened, a connection pool is created based on an exact matching algorithm that associates the pool with the connection string in the connection. Thus I decided to implement the method: SqlConnection.ClearPool(connection As SqlConnection) I placed this just before my END USING: Dim cur As Long = currentcounter * 100 / counter SetProgress(cur) SqlConnection.ClearPool(con) End Using
SQL Update not updating records in access
Trying to update the values of Comment and ProgressValue inside a table in access. The message box at the end pops up but none of the values are changed. Sub UpdateWeeklyReport() Dim con As OleDbConnection Dim com As OleDbCommand con = New OleDbConnection("provider=microsoft.jet.oledb.4.0;data source=C:\ProjectDatabase.mdb") com = New OleDbCommand("Update WeeklyReport Set Comment = #Comment, ProgressValue = #ProgressValue Where [EntryDate]='" & CBDate.SelectedValue.ToString & "' AND [AdminNo]=" & CBAdmin.SelectedValue & " AND [ClassCode]='" & CBClass.SelectedItem.ToString & "'", con) con.Open() com.Parameters.Add("#Comment", OleDbType.LongVarChar).Value = txtComment.Text com.Parameters.Add("#ProgressValue", OleDbType.Integer).Value = CBProgress.Text com.ExecuteNonQuery() con.Close() MessageBox.Show("Report Changed") intialconnection() End Sub End Class
Excel to Sql adding certain rows to sql server
cmd.CommandText = (Convert.ToString("SELECT * From [") & excelSheet) & "] Order By" & columnCompany & "OFFSET 0 ROWS FETCH NEXT 100000 ROWS ONLY " I am trying to map data form excel to an sql table but the excel table has a lot of records in it, so I run out of memory if I try to add all of the records at once. So what I am trying to do is use this line of code cmd.CommandText = (Convert.ToString("SELECT * From [") & excelSheet) & "] Order By B OFFSET 0 ROWS FETCH NEXT 100000 ROWS ONLY " So that 100000 row will be added. Then I intend to clear the data table and do the same only from 100001 to 200000 and so on. But I cant get this to work correctly, it works fine when it is just SELECT TOP 100000 FROM... but then I dont know how to get the next 100000 record. I think the Order By is where the issue is as I'm not sure how to get the correct column. Here is my code Below Private Sub btnMap_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnMap.Click 'Allows columns to be moved mapped in the correct order Dim nbColumnsToTransfer As Integer = dgvMapped.Columns.GetColumnCount(1) Dim indexes As List(Of Integer) = (From column As DataGridViewColumn In dgvLoadedata.Columns.Cast(Of DataGridViewColumn)() _ Take nbColumnsToTransfer _ Order By column.DisplayIndex _ Select column.Index).ToList() Dim columnCompany As Integer = indexes(0) Dim columnEmail As Integer = indexes(1) Dim columnFirstName As Integer = indexes(2) Dim columnSurname As Integer = indexes(3) Dim columnPostCode As Integer = indexes(4) Dim columnTelephone As Integer = indexes(5) Dim columnEmployeeBand As Integer = indexes(6) Dim DeleteConnection As New Data.SqlClient.SqlConnection DeleteConnection.ConnectionString = "Data Source=DC01\SQLEXPRESS;Initial Catalog=YCM;User ID=sa;Password=S0rtmypc!" DeleteConnection.Open() Dim sqlDTM As String = "Delete From TempMapped" Dim command As New SqlCommand(sqlDTM, DeleteConnection) command.ExecuteNonQuery() Dim sqlDTSR As String = "Delete From TempSplitReq" command = New SqlCommand(sqlDTSR, DeleteConnection) command.ExecuteNonQuery() DeleteConnection.Close() If chkContactSplitReq.Checked = True Then Dim dt As New DataTable() Using con As New OleDbConnection(conString) Using cmd As New OleDbCommand() Using oda As New OleDbDataAdapter() cmd.CommandText = (Convert.ToString("SELECT * From [") & excelSheet) & "] Order By" & columnCompany & "OFFSET 0 ROWS FETCH NEXT 100000 ROWS ONLY "" cmd.Connection = con con.Open() oda.SelectCommand = cmd oda.Fill(dt) con.Close() End Using End Using End Using Dim connection As String = "Data Source=DC01\SQLEXPRESS;Initial Catalog=YCM;User ID=sa;Password=S0rtmypc!" Using cn As New SqlConnection(connection) cn.Open() Using copy As New SqlBulkCopy(cn) copy.ColumnMappings.Add(columnCompany, 0) copy.ColumnMappings.Add(columnEmail, 1) copy.ColumnMappings.Add(columnFirstName, 2) copy.ColumnMappings.Add(columnPostCode, 3) copy.ColumnMappings.Add(columnTelephone, 4) copy.DestinationTableName = "TempSplitReq" copy.WriteToServer(dt) End Using End Using Dim SplitConnection As New Data.SqlClient.SqlConnection SplitConnection.ConnectionString = "Data Source=DC01\SQLEXPRESS;Initial Catalog=YCM;User ID=sa;Password=S0rtmypc!" SplitConnection.Open() Dim sqlSplit As String = "TempSplitReq_SaveToMapped" Dim commandSplit As New SqlCommand(sqlSplit, SplitConnection) commandSplit.ExecuteNonQuery() SplitConnection.Close() Else Dim dt As New DataTable() Using con As New OleDbConnection(conString) Using cmd As New OleDbCommand() Using oda As New OleDbDataAdapter() cmd.CommandText = (Convert.ToString("SELECT * From [") & excelSheet) + "] " cmd.Connection = con con.Open() oda.SelectCommand = cmd oda.Fill(dt) con.Close() End Using End Using End Using Dim connection As String = "Data Source=DC01\SQLEXPRESS;Initial Catalog=YCM;User ID=sa;Password=S0rtmypc!" Using cn As New SqlConnection(connection) cn.Open() Using copy As New SqlBulkCopy(cn) copy.ColumnMappings.Add(columnCompany, 0) copy.ColumnMappings.Add(columnEmail, 1) copy.ColumnMappings.Add(columnFirstName, 2) copy.ColumnMappings.Add(columnSurname, 3) copy.ColumnMappings.Add(columnPostCode, 4) copy.ColumnMappings.Add(columnTelephone, 5) copy.DestinationTableName = "TempMapped" copy.WriteToServer(dt) End Using End Using End If MsgBox("Data Mapping Complete") End Sub
A SqlBulkCopy object has a BatchSize property. I have a similar situation and set copy.BatchSize = 1000 And mine works with a large spreadsheet import. Adjust this property in your environment for better performance.
Populating Datagrid
I am creating a pageant scoring system, I have this functions for populating my datagridview's first column using query: Public Sub searchContestant() If comboCategory.SelectedIndex <> 0 Then Dim id As Integer = Login.JudgeBindingSource1.Current("Cont_id") Dim critID As Integer = Convert.ToInt32(lblID.Text) 'search the contest record by the contest_id Dim con As New SqlConnection Dim reader As SqlDataReader Dim reader2 As SqlDataReader Dim dt = New DataTable() Dim nc As New DataGridViewTextBoxColumn DataGridView1.Columns.Clear() DataGridView1.Rows.Clear() Try con.ConnectionString = "Data Source=BROWNIE\SQLEXPRESS;Initial Catalog=PSS;Integrated Security=True" Dim cmd As New SqlCommand("SELECT * from Criteria where Cat_id = '" & critID & "' ", con) Dim cmd2 As New SqlCommand("SELECT * from Contestant where Cont_id = '" & id.ToString() & "' ", con) con.Open() ' Execute Query reader = cmd.ExecuteReader() nc.Name = "Contestants" nc.Width = 250 nc.ReadOnly = True DataGridView1.Columns.Add(nc) While reader.Read() nc = New DataGridViewTextBoxColumn nc.Name = "(" & reader.GetString(2) & "%)" & reader.GetString(1) nc.HeaderText = "(" & reader.GetString(2) & "%)" & reader.GetString(1) nc.Width = 150 DataGridView1.Columns.Add(nc) End While Catch ex As Exception MessageBox.Show("Error while connecting to SQL Server." & ex.Message) Finally con.Close() 'Whether there is error or not. Close the connection. ' End Try Try 'populate contestant rows con.ConnectionString = "Data Source=BROWNIE\SQLEXPRESS;Initial Catalog=PSS;Integrated Security=True" Dim cmd2 As New SqlCommand("SELECT * from Contestant where Cont_id = '" & id.ToString() & "' ", con) con.Open() reader2 = cmd2.ExecuteReader() While reader2.Read() nc = New DataGridViewTextBoxColumn nc.Width = 100 DataGridView1.Rows.Add(reader2.GetString(2).ToString()) End While Catch ex As Exception MessageBox.Show("Error while connecting to SQL Server." & ex.Message) Finally con.Close() 'Whether there is error or not. Close the connection. ' End Try End If End Sub And for succeeding columns : Public Sub searchCriteria() Dim con As New SqlConnection Dim reader4 As SqlDataReader Dim dt = New DataTable() Dim nc As New DataGridViewTextBoxColumn DataGridView1.Columns.Clear() Try 'populate Criteria con.ConnectionString = "Data Source=BROWNIE\SQLEXPRESS;Initial Catalog=PSS;Integrated Security=True" Dim cmd As New SqlCommand("SELECT * from Criteria where Cat_id = '" & lblID.Text & "' ", con) con.Open() reader4 = cmd.ExecuteReader While reader4.Read nc = New DataGridViewTextBoxColumn nc.Name = reader4.GetString(1).ToString & Chr(13) & reader4.GetString(2).ToString & "%" nc.Width = 100 DataGridView1.Columns.Add(nc) End While Catch ex As Exception MessageBox.Show("Error while connecting to SQL Server." & ex.Message) Finally con.Close() 'Whether there is error or not. Close the connection. ' End Try End Sub This is the output: And everytime a judge login, I insert data into the system with this function: Public Sub makeTransaction() Dim con2 As New SqlConnection 'get the judge id upon logging in Dim judgeID = Login.JudgeBindingSource1.Current("Judge_id") 'get the contest id of the judge Me.JudgeTableAdapter.FillByJudgeID(Me.PSSDataSet.Judge, judgeID) Dim JudgeContestid = Login.JudgeBindingSource1.Current("Cont_id") Me.ContestantTableAdapter.FillByContestID(Me.PSSDataSet.Contestant, JudgeContestid) 'get the contestant id Me.ContestTableAdapter.FillByContestID(Me.PSSDataSet.Contest, JudgeContestid) Dim contestID As Integer = ContestBindingSource.Current("Cont_id") 'get the category of the contest Me.CategoryTableAdapter.FillByContestID(Me.PSSDataSet.Category, Convert.ToInt32(contestID)) For Each Category As DataRow In Me.PSSDataSet.Category.Rows Me.CriteriaTableAdapter.FillByCategoryID(Me.PSSDataSet.Criteria, Category("Cat_id")) For Each Contestant As DataRow In Me.PSSDataSet.Contestant.Rows For Each Criteria As DataRow In Me.PSSDataSet.Criteria.Rows TransactionsTableAdapter.Insert(JudgeContestid, Contestant("Con_id"), Criteria("Cri_id"), 0) Next Next Next End Sub Now my problem is, how can I populate the datagrid using the data recently made by MakeTransaction() function?