Performance issue with SQLite database with VB.NET - vb.net

Question

I am Inserting the data-table into SQLite Database. I am doing like this.
First I Fetch the data with getdata function and insert it into datatable, then with For Each Loop i made the Insert Command and Execute It. I am having 50000 Records it will take 30 Minutes to run.
Please Guide the suitable approach. Here is the Code.
Dim xtable As DataTable = getdata("select * from tablename")
Dim str As String = Nothing
For Each r As DataRow In xtable.Rows ''''HERE IT WILL TAKE TOO MUCH TIME
str = str & ("insert into tablename values(" & r.Item("srno") & "," & r.Item("name"));")
Next
EXECUTEcmd(str)
Public Function getdata(ByVal Query As String) As DataTable
connectionString()
Try
Dim mds As New DataTable
Dim mycommand As New SQLiteCommand(DBConn)
mycommand.CommandText = Query
Dim reader As SQLiteDataReader = mycommand.ExecuteReader()
mds.Load(reader)
Return mds
Catch ex As Exception
MsgBox("DB Error", vbCritical, "")
MsgBox(Err.Description)
Return Nothing
End Try
End Function
Public Sub EXECUTEcmd(ByVal selectcmd As String)
Using cn = New SQLiteConnection(conectionString)
cn.Open()
Using transaction = cn.BeginTransaction()
Using cmd = cn.CreateCommand()
cmd.CommandText = selectcmd
cmd.ExecuteNonQuery()
End Using
transaction.Commit()
End Using
cn.Close()
End Using
End Sub
here the Conncection String is:
conStr = "Data Source=" & dbpath & ";Version=3;Compress=True; UTF8Encoding=True; PRAGMA journal_mode=WAL; cache=shared;"

Answer 1

Use a stringbuilder to build your string, not string concatenation
Dim strB As StringBuilder = New StringBuilder(100 * 50000)
For Each r As DataRow In xtable.Rows
strB.AppendLine($"insert into tablename values({r.Item("srno")},{r.Item("name")});")
Next
Strings cannot be changed in .net. Every time you make a new string VB has to copy everything out of the old string into a new one and add the new bit you want. If each of your insert statements is 100 bytes, that means it copies 100 bytes, then adds 100, then copies 200 bytes and adds 100, then copies 300 bytes, then 400 bytes, then 500 bytes. By the time it has done 10 strings it has made 5.5 kilobytes of copying. By the time it's done 50 thousand strings it has copied 125 gigabytes of data. No wonder it's slow!
Always use a StringBuilder to build massive strings
--
I'm willing to overlook the sql injection hacking nag for this one, because of the nature of the task, but please read http://bobby-tables.com - you should never, ever concatenate values into an SQL as a way of making an sql that has some varying effect.
This entire exercise would be better done as this (pseudocode) kind of thing:
Dim sel as New SQLiteCommand("SELECT a, b FROM table", conn)
Dim ins as New SQLiteCommand("INSERT INTO table VALUES(:a, :b)", conn)
ins.Parameters.Add("a" ...)
ins.Parameters.Add("b" ...)
Dim r = sel.ExecuteReader()
While r.Read()
ins.Parameters("a") = r.GetString(0)
ins.Parameters("b") = r.GetString(1)
ins.ExecuteNonQuery()
End While
That is to say, you minimize your memory by reading rows one at a time out of ther edaer and inserting them one at a time in the insert; the insert command is prepared once, you just change the parameter values, execute it, change them again, execute it ... It's what parameterized queries were designed for (as well as stopping your app getting hacked when someone puts SQL in your variable, or even just stopping it crashing when you have an person named O'Grady

Answer 2

Maybe you must refactor your code like this:
Dim xtable As DataTable = getdata("select * from tablename")
Using cn = New SQLiteConnection(conectionString)
cn.Open()
Using transaction = cn.BeginTransaction()
Try
Using cmd = cn.CreateCommand()
cmd.Transaction = transaction
For Each r As DataRow In xtable.Rows ''''HERE IT WILL TAKE TOO MUCH TIME
cmd.CommandText = "insert into tablename values(" & r.Item("srno") & "," & r.Item("name") & ")"
cmd.ExecuteNonQuery()
Next
End Using
transaction.Commit()
Catch ex As Exception
transaction.Rollback()
End Try
End Using
End Using
Public Function getdata(ByVal Query As String) As DataTable
connectionString()
Try
Dim mds As New DataTable
Dim mycommand As New SQLiteCommand(DBConn)
mycommand.CommandText = Query
Dim reader As SQLiteDataReader = mycommand.ExecuteReader()
mds.Load(reader)
Return mds
Catch ex As Exception
MsgBox("DB Error", vbCritical, "")
MsgBox(Err.Description)
Return Nothing
End Try
End Function
Instead of concatenate an possible giant string, wrap all your inserts into a single transaction, like above. This will reduce the memory used and also make sqlite perform faster.