I'm not sure if this is the right approach but what I'm trying to do is create a queue that constantly runs so that I can keep adding things to it. Basically I would like to be able to add things to the queue and then it process it on a first come first served basis. I have the code below:
Namespace Managers
Public Class SQLQueueManager
Public Shared Sqlitems As New Queue
Public Sub StartProcessing()
Dim t1 As New Threading.Thread(AddressOf Process)
t1.Start()
End Sub
Public Shared Sub Process()
Do
SyncLock Sqlitems.SyncRoot
If Sqlitems.Count > 0 Then
SqlManager.ExecNonQuery(Sqlitems.Peek)
Sqlitems.Dequeue()
End If
End SyncLock
Loop
End Sub
End Class
End Namespace
i start this queue off using the following:
Dim sqlT As Thread
sqlT = New Thread(AddressOf SQLQueueManager.Process)
sqlT.Start()
and i add items to the queue using:
SQLQueueManager.Sqlitems.Enqueue(...)
Thanks
A custom approach like this can work. Consider, perhaps, using ThreadPool.QueueUserWorkItem :
ThreadPool.QueueUserWorkItem -- MSDN
Related
I have the need to process millions of files. Currently, I use a custom thread manager to do the work by using a DataGridView to keep track of the threads and a timer to check if more threads can start; kind of like (sudo):
Private Sub ThreadManager()
If AVailableThreads > 0 then
Dim t as Threading.Thread = New Thread(AddressOf MyThread)
t.Start()
AvailableThreads = AvailableThreads - 1
ThreadManager()
End If
End Sub
This has many drawbacks, the main ones being that the CPU and memory usage is high as each of the above threads process a full directory instead of each file independently.
So I have rewritten the process. Now I have a class that will perform the process at the file level and report back to the main thread the results; like so:
Imports System.IO
Public Class ImportFile
Public Class ImportFile_state
Public ID as Long = Nothing
Public FilePath as String = Nothing
Public Result as Boolean = False
End Class
Public Event ReportState(ByVal state as ImportFile_state)
Dim _state as ImportFile_state = New ImportFile_State
Public Sub New(ByVal ID as Long, ByVal FilePath as String)
MyBase.New()
_state.ID = ID
_state.FilePath = FilePath
End Sub
Public Sub GetInfo()
'Do the work here, but just return the result for this demonstration
Try
_state.Result = True
Catch ex As Exception
_state.Result = False
Finally
RaiseEvent ReportState(_state)
End Try
End Sub
End Class
The above class works like a charm and is very fast, uses almost no memory and next to nothing of the CPU. Albeit that I have only been able to test this with a few hundred threads using the Threading.Thread process.
Now I would like to use the ThreadPool.QueueUserWorkItem to execute the above class for each file allowing the system to control the number of threads to have running at any given time. However, I know I cannot just dump several million threads into the ThreadPool without locking up my server. I have done a lot of research on this and I have only been able to find examples/discussions on using the ThreadPool.QueueUserWorkItem for a few threads. What I need is to fire off millions of these threads.
So, I have two questions: 1) Should I even be trying to use the ThreadPool.QueueUserWorkItem to run this many threads, and 2) is the code below sufficient to perform this process without locking up my server?
Here is my code so far:
For Each subdir As String In Directory.GetDirectories(DirPath)
For Each fl In Directory.GetFiles(subdir)
'MsgBox(fl)
Dim f As ImportFile = New ImportFile(0, fl)
AddHandler f.ReportState, AddressOf GetResult
ThreadPool.QueueUserWorkItem(New Threading.WaitCallback(AddressOf f.GetInfo))
ThreadPool.GetAvailableThreads(worker, io)
Do While (worker) <= 0
Thread.Sleep(5000)
ThreadPool.GetAvailableThreads(worker, io)
Loop
Next
Next
Background:
I have a program that is processing lots of database records, and generating tasks to do. (In this case creating user accounts in AD).
Part of this is to create the user directories, for profiles and home directories, and setting the permissions on them.
This needs to wait until the ad account has replicated across all of our DC's.
So, my program will have a separate thread responsible for creating the directories, that will process a queue populated from the main thread.
I've done some research on Threading and come up with the following code pattern:
Imports System.Threading
Public Class Form1
Dim worker As Object
Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
worker = New workerObj(AddressOf resultcallback)
Dim t As New Thread(AddressOf worker.mainloop)
End Sub
Public Sub resultcallback(ByVal item As String)
Outbox.AppendText(item)
End Sub
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
worker.addItem(inbox.Text)
End Sub
End Class
Public Delegate Sub resultcallback(ByVal item As String)
Public Class workerObj
Private myQueue As New Queue(Of String)
Private myCallback As resultcallback
Dim item As String = "nout"
Public Sub New(ByVal callbackdelegate As resultcallback)
myCallback = callbackdelegate
End Sub
Public Sub mainloop()
While True
If myQueue.Count > 0 Then
item = myQueue.Dequeue()
myCallBack(item)
End If
Thread.Sleep(5000)
End While
End Sub
Public Sub addItem(ByVal item As String)
myQueue.Enqueue(item)
End Sub
End Class
Problem:
On the line Dim t as new Thread.....
Error 1 Overload resolution failed because no accessible 'New' is most specific for these arguments:
'Public Sub New(start As System.Threading.ParameterizedThreadStart)': Not most specific.
'Public Sub New(start As System.Threading.ThreadStart)': Not most specific. n:\visual studio 2013\Projects\ThreadTest\ThreadTest\Form1.vb 7 13 ThreadTest
Can anyone help tell me where I have gone wrong?
Cheers.
Threads do not have a public constructor, you need to call Thread.Start. I'd suggest you don't do that though. Writing thread-safe code is tricky enough when you do know about multithreaded programming.
Eg in your code you modify a Queue from two different threads without locking. Queue isn't thread safe and you can corrupt the queue. You should lock access to it or use ConcurrentQueue which is thread-safe. Another error is trying to modify a TextBox from another thread - this will lead to an Exception because only the UI thread is allowed to modify UI controls.
A better option though is to use the ActionBlock class from the DataFlow library which already does what you want: queue requests and process them in one or more separate threads.
Your code can be as simple as this:
Dim myFileWorker=New ActionBlock(Of string)(Function(path) =>DoSomething(path))
For Each somePath As String in ListWithManyPaths
myFileWorker.Post(somePath)
Next somePath
myFileWorker.Complete()
myFileWorker.Completion.Wait()
By default only one path will be processed at a time. To process multiple paths you pass an ExecutionDataflowBlockOptions object with the desired MaxDegreeOfParallelism:
Dim options=New ExecutionDataflowBlockOptions() With { .MaxDegreeOfParallelism=5}
Dim myFileWorker=New ActionBlock(Of String) Function(path) DoSomething(path),options)
Can someone help me understand how synclock statements work? I know there are some other posts about synclock here but I was not able to follow the answers very well. I am trying to get a simple multithreading example up and running but am having trouble. At first I tried the following code.
Public Class Class1
Public list As List(Of Integer) = New List(Of Integer)
Public Sub addInt(ByVal i As Integer)
SyncLock list
list.Add(i)
End SyncLock
End Sub
Public Sub MainLoop()
While list.Count < 50
SyncLock list
For Each i As Integer In list
Debug.WriteLine(i.ToString())
Next
End SyncLock
End While
End Sub
End Class
I had a simple winform with two buttons. I used the first button to create a obj of Class1 and start the MainLoop method in a new thread. And I used the seconded button to call the addInt method. However the code might work for a press or two and then lock up. After reading the other questions from this form I realized that the lockobject for the synclock statement did not work as I initially thought, and was just an arbitrary reference object that should not ever be changed. I think the syncLock statement just forces the whole code block to execute before passing processor control to a new thread. So I tried this using another lock object but now it just locks up.
Public Class Class1
Private padLock As String = "PADLOCK"
Public list As List(Of Integer) = New List(Of Integer)
Public Sub addInt(ByVal i As Integer)
SyncLock padLock
list.Add(i)
End SyncLock
End Sub
Public Sub MainLoop()
While list.Count < 50
SyncLock padLock
For Each i As Integer In list
Debug.WriteLine(i.ToString())
Next
End SyncLock
End While
End Sub
End Class
Can someone explain why the above code does not work? In a similar post someone mentioned that the Interlocked class would be useful but I could not figure out what that is or how that is used. A brief "Interlocked Class For Dummies" explanation would also be appreciated.
Ok I think I figured out the problem. I don't think my thread was deadlocking I think it was just starving my Application.Run() Thread of resources. When I added a Thread.Sleep(500) call after the End SyncLock in the MainLoop method the program worked as expected.
I'm trying to run a multi-threaded console app in VB and am having thread cross-over. Basically I want to run 5 threads, have them continually access a queue, process, and repeat until there's nothing left. When all threads have processed I want them to do something else. I'm attempting to use SyncLock to prevent multiple threads from accessing but it does not seem to be working. Any help would be appreciated!
Dim iThread As Integer
Dim manualEvents(4) As ManualResetEvent
Sub Main()
For i = 0 To 4
manualEvents(i) = New ManualResetEvent(False)
ThreadPool.QueueUserWorkItem(AddressOf DoOne)
Next
For Each handle As WaitHandle In manualEvents
handle.WaitOne()
Next
' do other stuff
EndSub
Private Sub DoOne()
Dim lockObject As New Object()
SyncLock (lockObject)
If QueryQueue.DoOne() Then
DoOne()
Else
manualEvents(iThread).Set()
iThread = iThread + 1
End If
End SyncLock
End Sub
The problem is with the locked resource, you're using lockObject as a synchronization lock resource which should be shared accros threads.
You have to make it an instance field.
Private Shared lockObject As New Object()
Private Sub DoOne()
SyncLock (lockObject)
If QueryQueue.DoOne() Then
DoOne()
Else
manualEvents(iThread).Set()
iThread = iThread + 1
End If
End SyncLock
End Sub
The problem is that you are creating and using a new instance of an object for locking on each thread. The naive solution is to promote lockObject from a local variable to class variable. That way each thread is using the same object to lock on. I say this is naive because you have exchanged one problem for another (albeit less severe). The new problem is that you have now made your parallel algorithm a serial algorithm since only one thread can being doing work at any given time.
The solution would be to lock access to the queue only while it is being changed. Then operate on the dequeued objects outside the lock so that the threads can perform work concurrently.
If .NET 4.0 is available to you could refactored the code like this.
Public Class Example
Private m_Queue As ConcurrentQueue(Of SomeObject) = New ConcurrentQueue(Of SomeObject)()
Public Shared Sub Main()
' Populate the queue here.
Dim finished = New CountdownEvent(1)
For i As Integer = 0 to 4
finsihed.AddCount()
ThreadPool.QueueUserWorkItem(AddressOf DoOne, finished)
Next
finished.Signal()
finished.Wait()
End Sub
Private Shared Sub DoOne(ByVal state As Object)
Try
Dim item As SomeObject = Nothing
Do While m_Queue.TryDequeue(item) Then
' Process the dequeued item here.
Loop
' There is nothing left so do something else now.
Finally
Dim finished = DirectCast(state, CountdownEvent)
finished.Signal()
End Try
End Sub
End Class
I used ConcurrentQueue to avoid having to use SyncLock entirely. I used CountdownEvent as a more scalable alternative to wait for work items to complete.
You need to share the same lockObject across the threads by making it an instance variable.
I'm having some trouble getting cross-thread communication/field-updating working properly in my VB.NET 2010 program. I'm trying to update a field on my main form whenever a thread that I've started throws an event. Here's a simplified version of my code:
My main form:
Public Class Main
' stuff
' Eventually, startProcessing gets called:
Private Sub startProcessing()
Dim processingClass = New MyProcessingClass("whatever")
AddHandler processingClass.processStatusUpdate, AddressOf handleProcessStatusUpdate
Dim processingThread = New Thread(AddressOf processingClass.process)
processingThread.Start()
End Sub
Private Sub handleProcessStatusUpdate(statusUpdate As String)
txtMainFormTextBox.Text = statusUpdate ' InvalidOperationException
' "Cross-threaded operation not valid: Control 'txtMainFormTextBox' accessed from a thread other than the thread it was created on"
End Sub
End Class
The class which raises the event:
Public Class MyProcessingClass
Private whatever As String
Public Event processStatusUpdate(status As String)
Public Sub New(inWhatever As String)
whatever = inWhatever
End Sub
Public Sub process()
' do some stuff
RaiseEvent processStatusUpdate(whatever)
End Sub
End Class
As you can see, the handler in my main class doesn't have access to the TextBox I need since it was triggered by a different thread (I think). I've tried a number of other approaches to get this working, including:
Moving the event handler to MyProcessingClass, and passing txtMainFormTextBox by reference (ByRef) to the class.
Having the actual thread start inside of MyProcessingClass instead of Main.
None of these have worked. Clearly there's a concept that I'm missing here. What's the best way to get this done? Thanks!
You need to update the textbox on the UI thread by calling BeginInvoke.
You should use the BackgroundWorker component, which does all of this for you.
Simply handle the DoWork and ProgressChanged events.