Getting "Database is Locked" when trying to move a list of records from one table to another table in SQLite - vb.net

I have a Public Sub to move a collection of records from one table to another in the same SQLite database. First it reads a record from strFromTable, then writes it to strToTable, then deletes the record from strFromTable. To speed things up, I've loaded the entire collection of records into a transaction. When the list involves moving a lot of image blobs, the db gets backed up, and throws the exception "The Database is Locked". I think what is happening is that it's not finished writing one record before it starts trying to write the next record. Since SQLite only allows one write at a time, it thows the "Locked" exception.
Here is the code that triggers the error when moving a lot of image blobs:
Using SQLconnect = New SQLiteConnection(strDbConnectionString)
SQLconnect.Open()
Using tr = SQLconnect.BeginTransaction()
Using SQLcommand = SQLconnect.CreateCommand
For Each itm As ListViewItem In lvcollection
SQLcommand.CommandText = $"INSERT INTO {strToTable} SELECT * FROM {strFromTable} WHERE id = {itm.Tag}; DELETE FROM {strFromTable} WHERE ID = {itm.Tag};"
SQLcommand.ExecuteNonQuery()
Next
End Using
tr.Commit()
End Using
End Using
When I get rid of the transaction, it executes without error:
Using SQLconnect = New SQLiteConnection(strDbConnectionString)
SQLconnect.Open()
Using SQLcommand = SQLconnect.CreateCommand
For Each itm As ListViewItem In lvcollection
SQLcommand.CommandText = $"INSERT INTO {strToTable} SELECT * FROM {strFromTable} WHERE id = {itm.Tag}; DELETE FROM {strFromTable} WHERE ID = {itm.Tag};"
SQLcommand.ExecuteNonQuery()
Next
End Using
End Using
I'm not very good with DB operations, so I'm sure there is something that needs improvement. Is there a way to make SQLite completely finish the previous INSERT before executing the next INSERT? How can I change my code to allow using a transaction?
Thank you for your help.
.

Ok ... here is the solution that I decided to go with. I hope this helps someone finding this in a search:
Dim arrIds(lvcollection.Count - 1) As String
Dim i as Integer = 0
' Load the array with all the Tags in the listViewCollection
For i = 0 to lvcollection.Count - 1
arrIds(i) = lvcollection(i).Tag 'item.Tag holds the Primary Key "id" field in the DB
Next
'build a comma-space separated string of all ids from the array of ids.
Dim strIds as String = String.Join(", ", arrIds)
Using SQLconnect = New SQLiteConnection(strDbConnectionString)
SQLconnect.Open()
Using tr = SQLconnect.BeginTransaction()
Using SQLcommand = SQLconnect.CreateCommand
SQLcommand.CommandText = $"INSERT INTO {strToTable} SELECT * FROM {strFromTable} WHERE id IN ({strIds});"
SQLcommand.ExecuteNonQuery()
SQLcommand.CommandText = $"DELETE FROM {strFromTable} WHERE ID IN ({strIds});"
SQLcommand.ExecuteNonQuery()
End Using
tr.Commit()
End Using
End Using
The IN statement allows me to pass all of the "id" values to be deleted as a batch. This solution is faster and more secure than doing them one by one with no transaction.
Thanks for the comments, and best wishes to everyone in their coding.

Related

Is there any SQL Statement procedure or code for optional requested field value?

I'm a beginner. I created a database in vb.net and I need to build a query, in the SQL Statement - Table Adapter, which returns records even if parameters are NULL in one or more textbox. To be clear, I have several textboxes (related to fields) with which I can filter record results and I want to refine my research as much as I fill textboxes, reverse if I fill just one of them randomly.
Sorry if I confused you, but I guess you get it anyway.
In its simplest form (assuming SQL server param concepts)
-- Define your columns to pull back/display
select t1.column1, t1.column2, t1.column3...
-- Define the table, give it an alias if you're using more than one or it has a silly name
from thetable t1
-- Apply filters
where
-- For each textbox/column search combo, do this...
(column1 = #field1 or #field1 is null)
or -- If the filter is restrictive, use AND here
(column2 = #field2 or #field2 is null)
or -- If the filter is restrictive, use AND here
...
I would dump the table adapter for this requirement.
I am building the sql string using a StringBuilder. StringBuilder objects are mutable, String is not.
To run this Code
1. I assumed Sql Server. If this is not the case change all the data object (Connectio and Command) to the proper provider.
Add your connection string to the constructor of the connection.
Add your table name where it says "YourTable"
I just used TextBox1 etc. as control names. Use your actual control names
Replace Field1, Field2 etc. with your actual column names.
The parameter names (by convention, they start with #) can be anything you want as long as they match the name you add to the Parameters collection.
You will have to check your database for the actual datatypes of the fields. Be sure to convert the TextBox values to the compatible type. TextBox.Text is a string so it will be compatible to .VarChar but note number types or dates.
I added a Debug.Print to check what the Sql string looks like. Be cautious about where I have spaces when building the string. You can see the result in the immediate window (available from Debug menu).
If you don't already have a DataGridView on your form, add one so you can see the reults of your query.
Finally, always use parameters, use Using...End Using blocks, and open your connection at the last minute.
Private Sub RunDynamicQuery()
Dim sb As New StringBuilder
Dim AndNeeded As Boolean
Dim dt As New DataTable
Using cn As New SqlConnection("Your connection string")
Using cmd As New SqlCommand
sb.Append("Select * From YourTable Where ")
If Not String.IsNullOrEmpty(TextBox1.Text) OrElse Not String.IsNullOrWhiteSpace(TextBox1.Text) Then
sb.Append("Field1 = #Field1")
cmd.Parameters.Add("#Field1", SqlDbType.Int).Value = CInt(TextBox1.Text)
AndNeeded = True
End If
If Not String.IsNullOrEmpty(TextBox2.Text) OrElse Not String.IsNullOrWhiteSpace(TextBox2.Text) Then
If AndNeeded Then
sb.Append(" And")
End If
sb.Append(" Field2 = #Field2")
cmd.Parameters.Add("#Field2", SqlDbType.VarChar).Value = TextBox2.Text
AndNeeded = True
End If
If Not String.IsNullOrEmpty(TextBox3.Text) OrElse Not String.IsNullOrWhiteSpace(TextBox3.Text) Then
If AndNeeded Then
sb.Append(" And")
End If
sb.Append(" Field3 = #Field3")
cmd.Parameters.Add("#Field3", SqlDbType.VarChar).Value = TextBox3.Text
AndNeeded = True
End If
sb.Append(";")
cmd.Connection = cn
Debug.Print(sb.ToString)
cmd.CommandText = sb.ToString
cn.Open()
dt.Load(cmd.ExecuteReader)
End Using
End Using
DataGridView1.DataSource = dt
End Sub

SqlDataAdapter.Update() of 2 million records is extremely slow

I have a customer that wants to import his sub-customers pricetools (more that 2.000.000 records) every day into a SQL Server database (and yeah....there are more than 900.000 rows of changes every day).
The data is provided in CSV format (not in RFC-4180 standard ç_ç, but nvm) and can be an Insert, Delete or Update of data.
My problem is that the insert of the data inside the database take more than 1 night to end and I need to speed it up.
What I'm doing at the moment is:
Cast the csv file into a Datatable (Tab1) (~3 minutes)
Select all data inside the previous table (Tab0) and match them with the Tab1 (~15 minutes, the unchanged rows are flagged as unmodified, so they are ignored in the adapter.Update, I check that thing for the first rows and seems that it works, I use dataRowToProcess.AcceptChanges() to achieve that).
Launch the following command to apply the changes (More than 5 hours for 900.000 changes):
cmdSQL = New SqlCommand(superQuery, cn)
Dim adapter As SqlDataAdapter = New SqlDataAdapter(cmdSQL)
adapter.MissingSchemaAction = MissingSchemaAction.AddWithKey
Dim build As SqlCommandBuilder = New SqlCommandBuilder(adapter)
build.SetAllValues = False
adapter.Update(dataTableCustomersDetail) 'Insert/Update records
If I have many inserts the process, it is slower than the same amount of updates.
What am I doing wrong? Am I missing some SqlDataAdapter option?
Thanks
Thanks to #matsnow i figuredOut a solution with SqlBulkCopy. Considering that half of the table change everitime and that is a static anag i decide that a Delete/Insert of the data is the fastest way to follow (Now it takes 5-6 Minutes instead of 10).
Code:
'Delete all table content
Dim cmd As SqlCommand = New SqlCommand("TRUNCATE TABLE " + tableName, cn)
cmd.ExecuteNonQuery()
'Insert all records
Using sbc As SqlBulkCopy = New SqlBulkCopy(cn)
sbc.DestinationTableName = tableName
sbc.BulkCopyTimeout = 1000
For Each column As DataColumn In dataTableCustomersDetail.Columns
sbc.ColumnMappings.Add(column.ToString(), column.ToString())
Next
sbc.WriteToServer(dataTableCustomersDetail)
End Using
Use Connection.BeginTransaction() to speed up the DataAdapter update.
cn.Open() 'open connection
Dim myTrans As SQLTransaction
myTrans = cn.BeginTransaction()
'Associate the transaction with the select command object of the DataAdapter
adapter.SelectCommand.Transaction = myTrans
adapter.Update(dataTableCustomersDetail) 'do the update as before
Try
myTrans.Commit()
Catch ex As Exception
myTrans.Rollback()
End Try
cn.Close()
With 8000 rows this changes the update time from over 5 minutes to 2 seconds

List View Population Errors

I have 3 columns that need populating when a user presses 'search' however every time that I click 'search' only the Employee ID appears, neither the 'First Name' nor the 'Last Name' are present in the List View. The data does exist in my Access Database, this is proved as the program produces a blank record instead of a null value error. The code that I am using to populate the List View is:
ds.Clear()
lstClockin.Items.Clear()
con.ConnectionString = provider & datafile
con.Open() 'Open connection to the database
sqlstatement = "SELECT * FROM [EmployeeAccounts]"
da = New OleDb.OleDbDataAdapter(sqlstatement, con)
da.Fill(ds, "allmembers") 'Fill the data adapter
con.Close()
Dim recordCount, x As Short
recordCount = 0
x = 0
recordCount = ds.Tables("allmembers").Rows.Count
With ds.Tables("allmembers")
Do Until x = recordCount
lstClockin.Items.Add(.Rows(x).Item(0))
lstClockin.Items(x).SubItems.Add(.Rows(x).Item(1))
lstClockin.Items(x).SubItems.Add(.Rows(x).Item(2))
lstClockin.Items(x).SubItems.Add(.Rows(x).Item(3))
x = x + 1
Loop
End With
The first 3 columns in the Database are, [Employee ID], [First Name] & [Last Name]
Any suggestions are welcome; however I have ruled out using a DataGridView or any control. As this program needs to use a ListView. Thankyou in advance!
There are several things that can be improved in the code:
Dim SQL = "SELECT Id, Name, Fish FROM Sample"
Using dbcon As New OleDbConnection(ACEConnStr)
Using cmd As New OleDbCommand(SQL, dbcon)
dbcon.Open()
Dim lvi As ListViewItem
myLV.SuspendLayout()
Using rdr = cmd.ExecuteReader
Do While rdr.Read
lvi = New ListViewItem(rdr.GetInt32(0).ToString)
If rdr.IsDBNull(1) Then
lvi.SubItems.Add("")
Else
lvi.SubItems.Add(rdr.GetString(1))
End If
If rdr.IsDBNull(2) Then
lvi.SubItems.Add("")
Else
lvi.SubItems.Add(rdr.GetString(2))
End If
myLV.Items.Add(lvi)
Loop
End Using
myLV.ResumeLayout()
End Using
End Using
Connections and other DB Provider objects allocate resources which need to be released or your app will leak. Using blocks for things that implement Dispose will close and dispose of them for you
There is no need for an DataAdapter, DataSet and DataTable since you are copying the data to the control. This code uses a DataReader to get the data.
Rather than SELECT * the query specifies the columns/order so it can use the Getxxxxx methods to get typed data. That doesnt matter a great deal in this case because everything gets converted to string for the ListView. lvi.SubItems.Add(rdr(COLUMN_NAME).ToString()) would also work.
It seems unlikely the ID column could be null, so the code only checks the other 2 for DbNull (another thing the DGV can handle without help).
Since the ListView is suboptimal and slow in adding items, SuspendLayout and ResumeLayout are used to minimize paints while it is populated.
I am not at all sure what ...the program produces a blank record means, but in order use a ListView like it is a grid, the View property must be Details and you have to have added 3 columns in the IDE (or manually create them in code). Nothing will show without those settings.
If the DataTable is needed/used elsewhere, you can still fill one without a DataAdpater and populate the LV from it:
...
dt.Load(cmd.ExecuteReader)
For Each row As DataRow In dt.Rows
lvi = New ListViewItem(row(0).ToString())
If DBNull.Value.Equals(row(1)) Then
lvi.SubItems.Add("")
Else
lvi.SubItems.Add(row(1).ToString())
End If
If DBNull.Value.Equals(row(2)) Then
lvi.SubItems.Add("")
Else
lvi.SubItems.Add(row(2).ToString())
End If
myLV.Items.Add(lvi)
Next
This uses a different DBNull check since it is using a DataRow and not the DataReader.

How do I query SQL data then insert or update depending on the result

I am a beginner at this. But let me explain what I need to do and show you my code
I have a CSV file.
inside the CSV I have a projectnumber, city,state,country
I have a SQL table with the same column
I want to use vb.net to check if projectnumber exists in sql table
if exists then I want to run update statement.
if it does not exists then I want to run insert statement.
I have the program working . but I am just wondering if this would be the correct way or my code is some hack way of doing it.
LEGEND:
DTTable is data table with CSV inside
DT is data table with SQL result data
First I fill insert all lines in the CSV into a data table
Dim parser As New FileIO.TextFieldParser(sRemoteAccessFolder & "text.csv")
parser.Delimiters = New String() {","}
parser.ReadLine()
Do Until parser.EndOfData = True
DTTable.Rows.Add(parser.ReadFields())
Loop
parser.Close()
then I use oledbdataadapter to run the select query and fill another data table with the result of the select statement
SQLString = "select * from tblProjects where ProjectID='" & DTTable.Rows.Item(i).Item("ProjectNumber") & "'"
da = New OleDb.OleDbDataAdapter(SQLString, Conn)
da.Fill(dt)
then I run if statement
If dt.Rows.Count = 0 then
SQLString = "INSERT STATEMENT HERE"
oCmd = New OleDb.OleDbCommand(SQLString, Conn)
oCmd.ExecuteNonQuery()
Else
SQLString = "UPDATE STATEMENT HERE"
oCmd = New OleDb.OleDbCommand(SQLString, Conn)
oCmd.ExecuteNonQuery()
End if
ALL above code is run inside a for loop, to go through all the lines in the CSV
For i = 0 To DTTable.Rows.Count - 1
what do you think?
please advise
thank you
Personally, I wouldn't use .NET. I would import the table into a temp SQL Server table and then write my queries to insert/update data from the temp table to the regular table. This is certainly the way you want to go if the dataset is large.
If this is a process you need to repeat frequently, you could make an SSIS package.
I'd run the select query using datareader = command.ExecuteReader(). Then:
If datareader.Read() then
'Update query using datareader(0) as a where predicate goes here
ElseIf datareader(0) = Nothing then
'Insert query goes here
End If
I should say, I'm a relative novice too though, so maybe others can suggest a more elegant way of doing it.

Concurrency violation updating a SQL database with a dataadapter

I'm having some trouble updating changes I made to a datatable via a dataadapter. I am getting "Concurrency violation: the UpdateCommand affected 0 of 10 rows"
'Get data
Dim Docs_DistributedTable As New DataTable("Docs_Distributed")
Dim sql = "SELECT DISTINCT CompanyID, SortKey, OutputFileID, SequenceNo, DeliveredDate, IsDeliveryCodeCounted, USPS_Scanned FROM Docs_Distributed_Test"
Using sqlCmd As New SqlCommand(sql, conn)
sqlCmd.CommandType = CommandType.Text
Docs_DistributedTable.Load(sqlCmd.ExecuteReader)
End Using
'Make various updates to some records in DataTable.
'Update the Database
Dim sql As String = "UPDATE Docs_Distributed "
sql += "SET DeliveredDate = #DeliveredDate "
sql += "WHERE SequenceNo = #SequenceNo"
Using transaction As SqlTransaction = conn.BeginTransaction("ProcessConfirm")
Try
Using da As New SqlDataAdapter
da.UpdateCommand = conn.CreateCommand()
da.UpdateCommand.Transaction = transaction
da.UpdateCommand.CommandText = sql
da.UpdateCommand.Parameters.Add("#DeliveredDate", SqlDbType.DateTime).SourceColumn = "DeliveredDate"
da.UpdateCommand.Parameters.Add("#SequenceNo", SqlDbType.Int).SourceColumn = "SequenceNo"
da.ContinueUpdateOnError = False
da.Update(Docs_DistributedTable)
End Using
transaction.Commit()
Catch ex As Exception
transaction.Rollback()
End Try
End Using
Now here's the catch. I am selecting DISTINCT records and essentially getting one row per SequenceNo. There may be many rows with the same SequenceNo, and I am hoping this will update them all. I'm not sure if this is related to my problem or not.
Your select is from "Docs_Distributed_Test" and your update is to "Docs_Distributed" - this may be the cause of your issue. Are the sequence ID's the same? (If not then perhaps it is indeed affecting 0 rows with it's update).
Other than that, you can always disable optimistic concurrency on your table-adapter and it will no longer enforce the validation (Though in this case that would likely result in no error but not updating any rows).
I don't understand the Microsoft-specific aspects of this, plus VB is often hard to follow. But this sequence seems suspect:
Using transaction As SqlTransaction = conn.BeginTransaction("ProcessConfirm")
Try
Using da As New SqlDataAdapter
da.UpdateCommand = conn.CreateCommand()
da.UpdateCommand.Transaction = transaction
conn.BeginTransaction is followed by conn.CreateCommand(). Isn't that a) useless, b) hazardous to the connection state, or c) potentially a race condition?