Spawning more than one thread in vb.net - vb.net

Question

I have an application that trawls for files on a network and then uploads them to a cloud service. It's not the fastest thing so I started to wonder whether I should finally try to get my head around threading.
What I want to do is have one thread that crawls the network looking for files to process and adding them to a queue, and then a number of uploader threads that read the queue and if there's work to do, they pull the task off the queue.
The reason for having more than one thread handling the uploading is that this part of the process is slow and a whole batch is likely to take days or even weeks based on initial tests. At the moment and without any threading, I have routine that finds all the files to upload and then an uploader routine uploads them.
So I'm assuming that if I can somehow have a single queue/list that is accessible to all threads, the thread that adds files to the list can be doing that whilst multiple uploader threads are pulling them off the list and processing them individually. It sounds like it should be possible but I can't see how. BTW most of the 'Sleep' statements in this test code are purely there for test purposes.
I found this sample which was helpful: https://cjhaas.com/2009/06/25/creating-a-simple-multi-threaded-vb-net-application/
And I based my test project on that.
The code below is complete but you will need to create a form with a progress bar, button and the labels. It builds and works fine and includes 3 threads, one for adding content to the queue (dummy data at the moment), one that processes the queue (just has a timer for now) and one to monitor the progress and update a progress bar.
I'm pleased that I got this far as I'M A COMPLETE NOVICE, but I can't work out what I would need to do to this code to make it spawn more that one worker thread AND be able to share the same queue; I tried peeling the queue out as a separate class but there were issues getting the thread to communicate with the queue class (I failed to make a note of the problems otherwise I would have added them for completeness).
Are there some easy changes that could be made to this to allow me to have more than one uploading worker thread? Or is there a better example I should look to adapt for my needs?
Option Explicit On
Option Strict On
Imports System.Threading
Public Class Form1
Private MonitorThread As Thread
Private WorkerThread As Thread
Private QueueThread As Thread
Private W As Worker
Private Delegate Sub UpdateUIDelegate()
Private Delegate Sub WorkerDoneDelegate()
Private Sub Monitor()
Do While WorkerThread.ThreadState <> ThreadState.Stopped 'Loop until the Worker thread (and thus the Worker object's Start() method) is done
UpdateUI() 'Update the progress bar with the current value
Thread.Sleep(250) 'Sleep the monitor thread otherwise we'll be wasting CPU cycles updating the progress bar a million times a second
Loop
WorkerDone() 'If we're here, the worker object is done, call a method to do some cleanup
End Sub
Private Sub UpdateUI()
If Me.InvokeRequired Then 'See if we need to cross threads
Me.Invoke(New UpdateUIDelegate(AddressOf UpdateUI), New Object() {}) 'If so, have the UI thread call this method for us
Else
'Me.ProgressBar1.Value = curIndex 'Otherwise just update the progress bar
'Me.ProgressBar1.Value = W.CurRun 'Otherwise just update the progress bar
Me.ProgressBar1.Maximum = W.Qtotal
Me.ProgressBar1.Value = W.Qtotal - W.Remaining 'Otherwise just update the progress bar
Me.Lbl_CurrentRun.Text = (W.Qtotal - W.Remaining).ToString
Me.Lbl_Total.Text = "Total = " + W.Qtotal.ToString
Me.Lbl_Remaining.Text = "Remaining = " + W.Remaining.ToString
Me.Lbl_Date.Text = "Date = " + W.MSGdate.ToLongTimeString
Me.Lbl_Path.Text = "Message = " + W.MSGpath
End If
End Sub
Private Sub WorkerDone()
If Me.InvokeRequired Then 'See if we need to cross threads
Me.Invoke(New WorkerDoneDelegate(AddressOf WorkerDone)) 'If so, have the UI thread call this method for us
Else
Me.Button1.Enabled = True 'Otherwise just update the button
Me.ProgressBar1.Value = W.Qtotal - W.Remaining ' Update these to ensure they show the final data
Me.Lbl_CurrentRun.Text = (W.Qtotal - W.Remaining).ToString
Me.Lbl_Remaining.Text = "Remaining = " + W.Remaining.ToString
End If
End Sub
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
Me.Button1.Enabled = False 'Disable the button
W = New Worker() 'Create our Worker object
QueueThread = New Thread(AddressOf W.PopulateQueue) 'Create a queue thread and tel it where to start when we call the start method
WorkerThread = New Thread(AddressOf W.Start) 'Create our Worker thread and tell it that when we start it it should call our Worker's Start() method
MonitorThread = New Thread(AddressOf Monitor) 'Create our Monitor thread and tell it that when we start it it should call this class's Monitor() method
QueueThread.Start() 'Start the queue thread which adds items to the queue
'Wait a while to allow some items to be added to the queue
System.Threading.Thread.Sleep(500)
WorkerThread.Start() 'Start the worker thread
MonitorThread.Start() 'Start the monitor thread which updates the progress bar
End Sub
End Class
Public Class Worker
Private iQtotal As Integer = 0 ' The total items processed
Private iRemaining As Integer ' Remaining items in the queue
Dim SortedList As New List(Of sMSG)()
Public ReadOnly Property Qtotal() As Integer
Get
Return Me.iQtotal
End Get
End Property
Public ReadOnly Property Remaining() As Integer
Get
'SortedList.TrimExcess()
Return SortedList.Count
End Get
End Property
Public ReadOnly Property MSGdate() As Date
Get
Return SortedList.FirstOrDefault.mDate
End Get
End Property
Public ReadOnly Property MSGpath() As String
Get
Return SortedList.FirstOrDefault.sFullpath
End Get
End Property
Public Sub PopulateQueue()
Dim tempMSG As New sMSG 'We will assemble the message in here before adding to the queue
tempMSG.sFullpath = "Path"
tempMSG.sLocation = "Location"
tempMSG.sLocationGUID = "GUID"
tempMSG.sLocationServerID = "666"
' This loop adds test data to the queue. It will be replaced with file system crawler
For i = 1 To 50
tempMSG.mDate = Date.Now
tempMSG.sFullpath = "Path " + i.ToString
SortedList.Add(tempMSG)
iQtotal = iQtotal + 1
Threading.Thread.Sleep(50)
SortList()
Next i
End Sub
Public Sub SortList()
SortedList = (From obj In SortedList Select obj Order By obj.mDate Descending).ToList()
End Sub
Public Sub New()
End Sub
Public Sub Start()
Dim nextMSG As sMSG
While SortedList.Count > 0
nextMSG = SortedList.FirstOrDefault 'Get the first item off the list
'' Do the upload of it...
System.Threading.Thread.Sleep(150)
SortedList.Remove(nextMSG) 'Remove it from the queue
End While
End Sub
End Class
Public Class sMSG
Public sFullpath As String
Public sLocation As String
Public sLocationGUID As String
Public sLocationServerID As String
Public mDate As Date
End Class
#######
So following the advice from Craig and djv I started to look at changing the code to use the ConcurrentBag object.
Unfortunately there is no way to remove items from the list/queue with the ConcurrentBag. Fortunately I came across the ConcurrentQueue and this works just fine.
The following code is not pretty but it works and in my tests I was able to see the performance improvements from the additional threads. It needs more work to perhaps manage how many threads are spawned but that's for later.
My thanks to by #Craig and #djv for trying to help and I hope this is of use to someone.
Option Explicit On
Option Strict On
Imports System.Threading
Imports System.Collections.Concurrent
Public Class Form1
Private MonitorThread As Thread
Private WorkerThread1 As Thread
Private WorkerThread2 As Thread
Private WorkerThread3 As Thread
Private QueueThread As Thread
Private Delegate Sub UpdateUIDelegate()
Private Delegate Sub BatchIsDOneDelegate()
Private tStart As Date
Private cq As New ConcurrentQueue(Of sMSG)
Public iQTotal As Integer
Private Sub Monitor()
Do While cq.Count > 0 'Loop until the queue is empty
UpdateUI() 'Update the progress bar with the current value
Thread.Sleep(250) 'Sleep the monitor thread otherwise we'll be wasting CPU cycles updating the progress bar a million times a second
Loop
BatchIsDOne() 'If we're here, the batch is done, call a method to do some cleanup
End Sub
Private Sub UpdateUI()
If Me.InvokeRequired Then 'See if we need to cross threads
Me.Invoke(New UpdateUIDelegate(AddressOf UpdateUI), New Object() {}) 'If so, have the UI thread call this method for us
Else
Me.ProgressBar1.Maximum = iQTotal
Me.ProgressBar1.Value = iQTotal - cq.Count 'Otherwise just update the progress bar
Me.Lbl_CurrentRun.Text = (iQTotal - cq.Count).ToString
Me.Lbl_Total.Text = "Total = " + iQTotal.ToString
Me.Lbl_Remaining.Text = "Remaining = " + cq.Count.ToString
Me.Lbl_Elapsed.Text = "Elapsed = " + (Date.Now - tStart).ToString
End If
End Sub
Private Sub BatchIsDOne()
If Me.InvokeRequired Then 'See if we need to cross threads
Me.Invoke(New BatchIsDOneDelegate(AddressOf BatchIsDOne)) 'If so, have the UI thread call this method for us
Else
Me.Button1.Enabled = True 'Otherwise just update the button
Me.ProgressBar1.Value = iQTotal - cq.Count ' Update these to ensure they show the final data
Me.Lbl_CurrentRun.Text = (iQTotal - cq.Count).ToString
Me.Lbl_Remaining.Text = "Remaining = " + cq.Count.ToString
End If
End Sub
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
Me.Button1.Enabled = False 'Disable the button
tStart = Date.Now 'Note the start time
QueueThread = New Thread(AddressOf PopulateQueue) 'Create a queue thread and tel it where to start when we call the start method
WorkerThread1 = New Thread(AddressOf ProcessNextMSG) 'Create 1st Worker thread and tell it that when we start it it should call our Worker's Start() method
WorkerThread2 = New Thread(AddressOf ProcessNextMSG) 'Create 2nd Worker thread and tell it that when we start it it should call our Worker's Start() method
WorkerThread3 = New Thread(AddressOf ProcessNextMSG) 'Create 2nd Worker thread and tell it that when we start it it should call our Worker's Start() method
MonitorThread = New Thread(AddressOf Monitor) 'Create our Monitor thread and tell it that when we start it it should call this class's Monitor() method
QueueThread.Start() 'Start the queue thread which adds items to the queue
System.Threading.Thread.Sleep(500) 'Wait a while to allow some items to be added to the queue just for test purposes
WorkerThread1.Start() 'Start the worker thread
WorkerThread2.Start() 'Start the worker thread
WorkerThread3.Start() 'Start the worker thread
MonitorThread.Start() 'Start the monitor thread which updates the progress bar
End Sub
Public Sub PopulateQueue()
Dim tempMSG As New sMSG 'We will assemble the message in here before adding to the queue
tempMSG.sFullpath = "Path"
tempMSG.sLocation = "Location"
tempMSG.sLocationGUID = "GUID"
tempMSG.sLocationServerID = "666"
' This loop adds test data to the queue. It will be replaced with a file system crawler
For i = 1 To 50
tempMSG.mDate = Date.Now
tempMSG.sFullpath = "Path " + i.ToString
cq.Enqueue(tempMSG) ' Add the MSG to the queue
iQTotal = iQTotal + 1
Threading.Thread.Sleep(50)
Next i
End Sub
Public Sub ProcessNextMSG()
Dim nextMSG As New sMSG
' While the queue is not empty, take a message from the top and process it
While cq.IsEmpty = False
If cq.TryDequeue(nextMSG) = False Then
' It failed to get next message from queue
Else
'' Do the uploading of it...
System.Threading.Thread.Sleep(350) 'Just to mimmick the uploading for now
End If
End While
End Sub
End Class
Public Class sMSG
Public sFullpath As String
Public sLocation As String
Public sLocationGUID As String
Public sLocationServerID As String
Public mDate As Date
End Class

Answer 1

I found that a ConcurrentQueue was what I needed and have pasted the full working code into the thread above. Thanks to both #Craig and #djv for trying to help.