I have a background worker where I currently am running a setup to loop through and send the http commands to these devices to update their firmware. Currently it's very slow, looping though each one and sending it.
My first question is, how can I multithread this so increase the speed? Usually I would make a task per device, add to a task array, and do a "wait all". But the device numbers could get up to 300 here so I want to limit it to a smart amount and do it dynamically.
And my second question is, is it wise to do so in the back groundworker that is already a thread (so I can show the progress bar)? Or is there another way to do it that is smarter?
Here's the current setup:
Private Sub BackgroundWorker1_DoWork(sender As Object, e As System.ComponentModel.DoWorkEventArgs) Handles BackgroundWorker.DoWork
'loop through devices
For Each assignment As FirmwareAssign In assignList
'firmwareassign is a struct with credentials for http, ip address, and device brand as strings
'Send Update Request
Select Case assignment.devcebrand.tolower
Case "brand1"
Logstatement += brand1.updateFirmware(assignment, assignment.deviceipaddress)
Case "brand2"
Logstatement += brand2.updateFirmware(assignment, assignment.deviceipaddress)
Case "brand3"
Logstatement += brand3.updateFirmware(assignment, assignment.deviceipaddress)
End Select
'don't worry, not real names of my classes
Next
End Sub
These functions are network based. They will grab the new firmware image, and send it to the ip address over the network. So I also want to limit it since we don't want the bandwidth to go nuts. Then it will add it to the logstatement (a string) to display at the end for the user to know if something went wrong.
Since you are sending these across the network and your bottle-neck is likely the network, running multiple threads will not speed it up, if anything it will slow down.
If this is not the case, or you want to test this anyway, here are your 2 answers:
1: If you are already doing a collection, in this case a task array, your are almost there. What I do is stick another loop in your current loop and run while the array/collection length is greater that X (maybe 10 in your case) and in that loop, check the thread status and remove any of those that are complete. This will make it jump out of the inner loop and add more threads until you hit X again.
2: While I don't think there is anything that prevents you from running threads from other threads, the logistics of getting updates back to the GUI might be a nightmare. I would spawn the threads from your main thread which makes progress updates real easy.
Related
I am working on a VB.NET Windows Forms application where the user is supposed to be able to determine how many processes the application is allowed to launch at a time.
My current method mostly works but I've noticed that occasionally the application goes over the set amount. I use two global variables for this, _ConcurrentRuns which is 0 at the start of the application, and _MaxConcurrentRuns which is set by the user.
Private _sync As new Object()
' This is called Synchronously
Private Function RunModel() As Boolean
If CancelExectuion Then Return CancelCleanup()
Do While True
SyncLock _sync
If _ConcurrentRuns < _MaxConcurrentRuns Then
Interlocked.Increment(_ConcurrentRuns)
Exit Do
End If
End SyncLock
Threading.Thread.Sleep(50)
Loop
'This is what will launch an individual process and close it when finished
ret = RunApplication(arg)
' The process has been closed so we decrement the concurrent runs
Interlocked.Decrement(_ConcurrentRuns)
Return ret
End Function
The goal is to let only one thread exit the while loop at a time, I'm not able to catch it in the debug mode however in the task manager it will occasionally go 1-3 processes over what it's supposed to use. This makes me assume that somehow multiple threads are getting inside the synclock somehow, but I have no clue how that could be happening.
I will be very grateful for any and all help that can be provided, thanks for taking the time to read my question.
So it appears that my solution works for this, I don't want to delete this question because it might be helpful to somebody else in the future.
Answer: Use better process monitoring software / set priority to high in task manager.
I am currently making a vb program that i plan to make very big. I have a decent knowledge of visual basic but today i came across something i do not understand. Because of the huge size of my program , i decided to try and keep the program as organized as possible by putting specific subs in modules. These subs consist of httprequest , webbrowsers(control), webclients and alot of loops. In order to prevent these subs from lagging my main application i thread them using threading.thread and i start them from my main form. But this leads to two problems.
Problem 1: The threads cannot in any way interact with the main form.
Once the a httprequest or webclient collects the information from my desired website, i am trying to make it add the info to a listbox in my main form, So what i did is it typed
Msgbox("Info Sent")
form1.listbox1.items.add(String)
The first messagebox will show but although the code right under it runs, nothing is added to the first forms listbox.I am not using delegates to transfer the information, instead, although its not a good habit, i am using checkforillegalcrossovers.
Problem 2: Threading with a webbrowser.
Threading with a webbrowser using threading.thread also does not work because it causes an active x error. After looking it up i found that a solution was to use a single threaded apartment but this would not work because i may need multiple threads running off the same sub at once.
One solution that i have found to this problem is creating another form completely and setting it invisible, and since the form is its own thread i do not need to use threading.thread , but the problem comes when i am trying to create multiple threads, or else i can somehow dynamically create the threads and put the subs inside of it programically this method wont work And even if it does i feel that it is sloppy so i will leave this for one of two last resorts.
The other solution is the most simple one in which i just put all of the code in the main form, but if i keep on doing that form1 is gonna become huge and sloppy, doing this wont solve the webbrowser problem either and even when using regions i still feel that something that 1000+ lines deserves its own class.
There must be some solution out there that solves these problems. Any help would be appreciated, Thanks.
I checked my code for updating the progress bar, and using a single thread with synclock will NOT work. They way I make it work is perform the step of the pbar each time after a thread is started as I have limited total threads (say less than 5 threads). Thus, even the progress bar steps before the threads are finished, but it will not progress further before new threads started. It is not 100% accurate but it more or less telling the progress
'update the progress bar
some_form.PBar1.PerformStep()
' This while loop is to count the existing running thread,
' and determine whether new thread should start
While 1
Dim t2 = New System.Threading.Thread(Sub() WaitForPermission())
t2.Start()
t2.Join()
If proceed_gen Then
Exit While
End If
End While
'Start doing what I need to do
Dim t1 = SomeSub()
t1.Start()
'End of code, as VB doest not have thread.detach()
Correct me if I am wrong, but you probably have to use a background worker. I know this is annoying, but this is the limitation of VB.net.
Or, you can have something like this (pseudo code, not tested)
structure some_struct
'define the strings you want to update, and their status such that
'main() knows if you need to update the stuff to the form
' You can also put how many threads are running, and the status of each thread,
'such that the main knows if all threads are completed
end structure
sub your_sub()
'Manipulate the website, and update the data structure with
'proper stuff you need
end sub
sub main(){
dim t1 = New System.Threading.Thread(Sub() your_sub())
t1.start()
' I am listing only one threads here, but start as many as you want
'check if there are strings that you need to update to the form
while 1
'check if there are any stuff you want to update from the data structure.
' Make sure you use synclock on the data structure, so each thread won't fight each other on accessing the data struct
dim should_update as boolean = false
should_update = 'Something thatyou should implement to judge whether you should update the form.
'You can start a thread and join it so the thread won't fight with other threads for accessing the data structure
dim some_string as string
if should_update
some_string = 'You may also need a thread to join to get the stuff you need. Dim the string as an array if it is required.
'You can also try pass by ref
'if you need to use thread to access the data structure to know if you need to update the form
form1.listbox1.items.add(some_string )
end if
end while
end sub
This is an ugly solution, but it will help you do the job...
I'm trying to fill a dataset, but I want to limit the amount of time the system has to fill this dataset to 30 seconds.
I have tried (as suggested elsewhere on SO):
Dim T As Date = Date.Now
da.Fill(ds, "DATASET")
Do
If (Date.Now - T).TotalSeconds >= 30 Then
Main.VIEW_Title.Text = "Error In Connection..."
Exit Sub
End If
Exit Do
Loop
But the system just hangs anyway during the da.Fill(ds, "DATASET") section and doesn't ever exectute the "Error In Connection" message. It doesn't matter if I put the line inside the DO either, because it stops there. What I need is for it to execute the fill command, and then if it doesn't complete in 30 seconds, to allow me to handle that error.
Thanks
Unfortunately, that's not so easy.
What you can do is:
Start a background thread and do the fill operation in that thread.
In the main thread, wait for 100ms, check if the new thread completed, repeat at most 300 times.
Now the real problem is: What do you do if the main thread determines that the background thread has not finished after 30 seconds? DataAdapter.Fill does not provide a method to gracefully interrupt the operation, so you'll have to let it continue in the background.
This also means that you must use a separate data connection in the background thread, since the ADO.NET classes are not thread-safe.
I'd rather suggest a different approach: A data selection operation taking more than 30 seconds implies that either:
you select too much data (there's really no use showing 1 million rows to the user) or
your query needs optimization (such as a well-placed index).
Thus, I suggest to fix the root cause instead of using complicated workarounds to cover-up the symptom.
I take a part on a project and after many tests, me and my colleagues feel little disappointed by our coding skills and methodology. The issue is the following:
The application must read data from some type of RS-485 devices connected on a single bus via one or multiple serial ports. However, these devices come from different manufactures so as a result, they have different connection settings, different command sets and return different data. Moreover, each serial port can handle none, one or multiple devices simultaneously.
Each of our custom COMPort objects contains a device list and each device contains the type and the device settings that are recommended. So, after loading all settings and building all objects, the application starts polling each device in each COMPort object list. Every time, before writing each command, the serial port buffers are emptied and its settings change (Baud-rate, Parity, Handshake) without closing and reopening the port. After every command, a Threading.Thread.Sleep(delay) is executed with variable delay, depending on the response time of each device (delay varies from 150ms to 800ms). After Sleep() ends, application Reads the serial port and gets expected data. After that, either if BytestoRead > 0 or BytestoRead = 0 or an Exception occurs(wrong data returned, error in checksums, etc.), this function returns and application continues to read the next device in the list.
So, application executes in an endless loop, reading all devices from start to end, until it finally ends when user stops it from UI (when the project is a Windows Forms) or when the service Stops (when the project is a Windows service). The application does not use DataReceived event because we faced a lots of trouble, so the Read procedure is done manually.
Finally, application has a global log file which is accessible from all objects and holds all usual and unusual events and each COMPort object has its own log file in which all valid read data are written. Moreover, each COMPort has a timer object which checks for daytime changes per second and everyday at midnight all log files close and new files are opened.
The main problem is that all this seems to work great but unfortunately the application stops writing to logs after 1-2 hours (most of the times after 1:58 minutes exactly on one client machine), then after some minutes it crashes, sometimes with Send or Don't send window and sometimes silently. The CPU load never exceeds 1% and private & virtual memory does not seem to have a leak because it doesn't grow.
I have to mention that this application is tested on 3 machines (Windows XP, Windows 2003), on the following situations: mixed types of devices, one type of device, no devices connected (using null modem cable on one development machine.) Even on the last situation, the application crashes silently. This leads me to the thought that the issue has nothing to do with the Read procedure because in the “no devices attached” scenario, there is no data to be read. Also, the project is built with VB.Net 2008 using .Net Framework 2.0.
EDIT : Below is a small summary of the Read, Write and Polling procedures. I wrote it on Notepad++ as an example because I don't have access to the real code right now, so excuse any syntax errors. The aim of this is to give a better explanation of the situation I describe above and nothing more.
Private deviceIndex as Integer = -1
Private deviceList as List(Of Devices)
Private serialPort as SerialPort()
Private Sub PollSensor()
Try
If (deviceIndex <> -1) And deviceIndex > deviceList.Count then
deviceIndex = 0
End If
With serialPort
If .IsOpen then
.DiscardInBuffer()
.DiscardOutBuffer()
' SerialPort attributes must change during runtime to suit device requirements.
' Some devices require for example 4800 BaudRate and some other 9600.
.BaudRate = deviceList(deviceIndex).BaudRate
.Parity = deviceList(deviceIndex).Parity
.Handshake = deviceList(deviceIndex).HandShake
' Write command to device, passing device specific response time.
' Some devices have for example 150ms response time and some other 800ms.
WritetoDevice(new Byte[] {01, 01}, deviceList(deviceIndex).ResponseTime)
End If
End With
Catch ex As Exception
' Log
End Try
End Sub
Private Sub WritetoDevice(ByVal command As Byte[], ByVal responseTime As Integer)
Try
serialPort.Write(command, 0, command.Length)
' Cause application to pause for time equal with the device response time.
Threading.Thread.Sleep(responseTime)
' If device response is regular, data may have been available in serial port.
If ReadDevice() = 0 then
' Poll next device
deviceIndex += 1
PollSensor()
End If
Catch ex As Exception
' Log
End Try
End Sub
Private Function ReadDevice() As Integer
Dim data as String = ""
Try
With serialPort
If .IsOpen then
If .BytesToRead() > 0 Then
Do
data += .ReadExisting()
Until .BytesToRead() = 0
' Do something with data
End If
End If
End With
Catch ex As Exception
' Log
Finally
' Continue with the next device in deviceList, ignoring the current device Read() result
' This happens because devices return faulty data too often and the polling must continue
Return 0
End Try
End Function
We have tried to use the Debug Diagnostics Tool 1.2 x86 but the results returned, do not help us to understand the source of the issue. Some of the exported reports contain the following, although all exceptions are handled in Try..Catch blocks:
- The following threads in ... are waiting in a WaitOne
- 6,25% of threads blocked
- System.ExecutionEngineException
- System.StackOverflowException
- System.OutOfMemoryException
- System.FormatException
- System.Threading.ThreadAbortException
Does anyone have a better idea on how the above scenario could work?
Feel free to criticize any of the above methods, because that will help us to become better.
I've read a lot of different questions on SO about multithreaded applications and how to split work up between them, but none really seem to fit what I need for this. Here's how my program currently basically works:
Module Module1
'string X declared out here
Sub Main()
'Start given number of threads of Main2()
End Sub
Sub Main2()
'Loops forever
'Call X = nextvalue(X), display info as needed
End Sub
Function nextvalue(byval Y as string)
'Determines the next Y in the sequence
End Function
End Module
This is only a rough outline of what actually happens in my code by the way.
My problem being that if multiple threads start running Main2(), they're dealing with the same X value as in the other threads. The loop inside of main2 executes multiple times per millisecond, so I can't just stagger the loops. There is often duplication of work done.
How can I properly divide up the work so that the two threads running simultaneously never have the same work to run?
You should synchronize the generation and storage of X so that the composite operation appears atomic to all threads.
Module Module1
Private X As String
Private LockObj As Object = New Object()
Private Sub Main2()
Do While True
' This will be used to store a snapshot of X that can be used safely by the current thread.
Dim copy As String
' Generate and store the next value atomically.
SyncLock LockObj
X = nextValue(X)
copy = X
End SyncLock
' Now you can perform operations against the local copy.
' Do not access X outside of the lock above.
Console.WriteLine(copy)
Loop
End Sub
End Module
A thread manager is required to manage the threads and the work that they do. Say it is desirable to split up the work into 10 threads.
Start the manager
Manager creates 10 threads
Assign work to the manager (queue up the work, let's say it queues up 10000 work items)
Manager assigns a work item to complete for each of the 10 threads.
As threads finish thier work, they report back to the manager that they are done and recieve another work item. The queue of work should be thread safe so that items can be enqueued and dequeued. The manager handles the management of work items. The threads just execute the work.
Once this is in place, work items should never be duplicated amongst threads.
Use a lock so that only one thread can access X at a time. Once one thread is done with it, another thread is able to use it. This will prevent two threads from calling nextvalue(x) with the same value.