How can I run two tasks concurrently? - vb.net

Question

I have this for/next loop where I download a file and then process and load its contnets into a data base:
For Each f As WinSCP.RemoteFileInfo In remotefilesinf
If DownloadFile(FTPSession, CacheDirPath, "/mnt/usb", f) Then
LoadDB(System.IO.Path.Combine(CacheDirPath, f.Name))
Else
MsgBox("Download failed.")
End If
Next
In order to speed things up, how can I do the DB loading while the next file is downloading? I cannot do the DBLoad until each file download is complete and I can only do one DBLoad task at a time due to locking of the database.
I tried using a background worker for the LoadDB task but the RunWorkerCompleted event will not fire while the UI thread is busy with the download so i do not know when I can do the next DBload (DB not locked).
Any advice appreciated.

Answer 1

Here is another try since the requirement for the question have changed:
Public Class Form1
Shared rnd As New Random
Private download_que As New Queue(Of String)
Private process_que As New Queue(Of String)
Private download_thread As Thread
Private process_thread As Thread
Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
End Sub
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
download_que.Enqueue("File 1.db")
download_que.Enqueue("File 2.db")
download_que.Enqueue("File 3.db")
download_que.Enqueue("File 4.db")
download_que.Enqueue("File 5.db")
download_que.Enqueue("File 6.db")
If download_thread Is Nothing then
download_thread = New Thread(AddressOf DownloadFiles)
download_thread.Start()
End If
End Sub
Private AppendTextCaller As New Action(Of TextBox, String)(AddressOf AppendText)
Public Sub AppendText(ByVal control As TextBox, ByVal text As String)
control.AppendText(text)
End Sub
Public Sub DownloadFiles()
Dim file As String
While download_que.Count > 0
SyncLock download_que
file = download_que.Dequeue()
End SyncLock
Dim path As String = Download(file)
SyncLock process_que
process_que.Enqueue(path)
End SyncLock
If process_thread Is Nothing Then
process_thread = New Thread(AddressOf ProcessFiles)
process_thread.Start()
End If
End While
download_thread = Nothing
End Sub
Public Sub ProcessFiles()
Dim path As String, ok As Boolean
ok = True
While process_que.Count > 0 And ok
SyncLock process_que
path = process_que.Dequeue()
End SyncLock
ok = LoadDB(path)
End While
process_thread = Nothing
End Sub
Public Function Download(ByVal filename As String) As String
Dim sw = Stopwatch.StartNew()
Me.Invoke(AppendTextCaller, TextBox1, filename)
Thread.Sleep(1500 + 500*rnd.Next(15))
Dim message As String = String.Format(" ({0:F1} sec)", sw.ElapsedMilliseconds / 1000)
Me.Invoke(AppendTextCaller, TextBox1, message)
Me.Invoke(AppendTextCaller, TextBox1, Environment.NewLine)
Return IO.Path.Combine(IO.Path.GetTempPath(), filename)
End Function
Public Function LoadDB(ByVal path As String) As Boolean
Dim sw = Stopwatch.StartNew()
Dim filename = IO.Path.GetFileName(path)
Me.Invoke(AppendTextCaller, TextBox2, filename)
Thread.Sleep(800 + 500*rnd.Next(6))
Dim message As String = String.Format(" ({0:F1} sec)", sw.ElapsedMilliseconds / 1000)
Me.Invoke(AppendTextCaller, TextBox2, message)
Me.Invoke(AppendTextCaller, TextBox2, Environment.NewLine)
Return True
End Function
End Class

Answer 2

What about using two backgroundworkers? Use one to download files, the other one to stuff them into the db.
If a download completes, append the file to a list and each time a db update finishes, look at that list from bgw2...

Answer 3

You can run the DBLoad on a thread and set a ManualResetEvent to stop the execution before lauching the new DBLoad thread until the other ona finished.
Dim locker as New ManualResetEvent(True)
The locker acts like a traffic light, stoping the execution and waiting when is marked and going throught when otherwise.
Block the locker anywhere with:
locker.Reset()
Unblock the locker anywhere with:
locker.Set()
Set a stoping spot:
locker.WaitOne()
To see full capabilities see MSDN.

Answer 4

I think this is what you want:
Public Function DownLoadFile(ByVal f As String) As String
Trace.WriteLine("Start Downloading " & f)
Dim x As Integer = ProgressBar1.Value
Threading.Thread.Sleep(2000)
Me.Invoke(SetProgressCaller, x + 25)
Trace.WriteLine("Done Downloading " & f)
Return IO.Path.Combine(IO.Path.GetTempPath(), f)
End Function
Public Sub LoadDB(ByVal f As String)
Trace.WriteLine("Start Loading " & f)
Dim x As Integer = ProgressBar1.Value
Threading.Thread.Sleep(1000)
Me.Invoke(SetProgressCaller, x + 25)
Trace.WriteLine("Done Loading " & f)
End Sub
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
ProgressBar1.Value = 0
Dim f_path_1 = DownLoadFile("File 1")
Dim t1 As New Threading.Thread(AddressOf LoadDB)
t1.Start(f_path_1)
ProgressBar1.Value = 50
Dim f_path_2 = DownLoadFile("File 2")
Dim t2 As New Threading.Thread(AddressOf LoadDB)
t2.Start(f_path_2)
End Sub
' Can be called using Form.Invoke() from any thread
Private SetProgressCaller As New Action(Of Integer)(AddressOf SetProgress)
' Set progress bar in main thread
Public Sub SetProgress(ByVal pct As Integer)
ProgressBar1.Value = pct
End Sub
with the results:
Start Downloading File 1
Done Downloading File 1
Start Downloading File 2
Start Loading C:\Users\#####\AppData\Local\Temp\File 1
Done Loading C:\Users\#####\AppData\Local\Temp\File 1
Done Downloading File 2
Start Loading C:\Users\#####\AppData\Local\Temp\File 2
Done Loading C:\Users\#####\AppData\Local\Temp\File 2
which translates to
Downloading of file 1 (takes 2 sec)
Loading file 1 into DB (takes 1 sec) AND
Downloading of file 2 (takes 2 sec)
Loading of file 2 into DB (takes 1 sec)