Working with write queues and multi threading - vb.net

I have a process running in a separate thread which reads data over Ethernet and raises an event for each read. Then, this data is processed over a tree of classes. At the top of my class tree, I want to store the processed data to SQL tables. This read rate can be rather fast and often at times but on average its slower than my write rate to SQL. This is why I want a queue. Also I want my window not to freeze, so I want a separate thread.
The code below works but I'm fairly inexperienced in multi threading, is this a good and robust way to accomplish my task?
Private addLogQueue As New Queue
Private dequeueing As Boolean = False
Public Sub addLogHandler(ByVal machineName As String, ByVal LogTime As Date, ByVal EventLog As String, ByVal EventValue As String)
addLogQueue.Enqueue("INSERT INTO ... some SQL CODE")
Dim Thread1 As New System.Threading.Thread(AddressOf addLog)
Thread1.Start
End Sub
Private Sub addLog()
If Not dequeueing Then
dequeueing = True
While addLogQueue.Count <> 0
Try
SQLCon.Open()
SQLCmd = New SqlCommand(addLogQueue.Dequeue(), SQLCon)
SQLCmd.ExecuteNonQuery()
SQLCon.Close()
Catch ex As Exception
MsgBox(ex.Message)
If SQLCon.State = ConnectionState.Open Then
SQLCon.Close()
End If
End Try
End While
dequeueing = False
End If
End Sub

I would refer you to the article: Beginners Guide to Threading. Also, for performance reasons you may want to keep the thread open, and the database connection open and simply use a wait event to determine if the queue should be checked for more entries. See this article: An Overview of Synchronization. Keeping the connection open would really depend on the frequency of arrival of messages. Very frequent: keep connection open, intermittent: Close after use.
Dim Thread1 As New System.Threading.Thread(AddressOf addLog)
Private Shared mre As New ManualResetEvent(False)
Private addLogQueue As New Queue
Private shutDown as boolean = false
Public Sub Main
Thread1.Start
End Sub
Public Sub addLogHandler(ByVal machineName As String, ByVal LogTime As Date, ByVal EventLog As String, ByVal EventValue As String)
addLogQueue.Enqueue("INSERT INTO ... some SQL CODE")
mre.Set()
End Sub
Private Sub addLog()
dim cmdString as String
Try
Using SQLCon as New SQLConnection
SQLCon.Open()
While Not shutDown
mre.WaitOne()
mre.Reset()
While addLogQueue.Count <> 0
cmdString = addLogQueue.Dequeue()
SQLCmd = New SqlCommand(cmdString, SQLCon)
SQLCmd.ExecuteNonQuery()
End While
End While
End Using
Catch ex As Exception
My.Application.Log.WriteException(ex,TraceEventType.Error, cmdString)
End Try
End Sub
Also, you may want to lookup a few articles on writing services as this sort of thing works very much better as a service, if you want the program to do unattended monitoring. Also, you could have one thread accepting the command strings and adding them to the queue, and the second thread writing them to the database.

Related

How to properly set up BackgroundWorker in WPF VB

I am working in VB and when the end user clicks on a button, I want to show a “Please wait” window (no progress update necessary) and trigger the process in the background on another thread as it might take a while to complete. I understand BackgroundWorker is the recommended approach for this, and although I have previously set up a BGW on a windows forms application, I am new to WPF and having a hard time understanding the differences.
In order to reuse some of my code, I have written a function (“DbOperation”) that executes a SQL stored procedure (from parameter) and returns a data table (or NULL if no output), and it seems to work well everywhere except in concert with my BGW.
I understand UI should only be affected by the main thread, and background threads should not touch the UI, and the BGW thread should call the long-running process.
Private Sub btnProcessReports_Click(sender As Object, e As RoutedEventArgs) Handles btnProcessReports.Click
'Set up background worker
bgw.WorkerReportsProgress = False
bgw.WorkerSupportsCancellation = False
AddHandler bgw.DoWork, AddressOf bgw_DoWork
AddHandler bgw.RunWorkerCompleted, AddressOf bgw_RunWorkerCompleted
'Show please wait window
f.Label1.Text = "Importing web reports. Please wait."
f.Show()
'Start the work!
bgw.RunWorkerAsync()
End Sub
Private Sub bgw_DoWork(sender As Object, e As DoWorkEventArgs)
'Set current thread as STA
Thread.CurrentThread.SetApartmentState(ApartmentState.STA)
'Trigger job
Dim Qry As String = "EXEC [msdb].[dbo].sp_start_job N'Import Online Payment Portal Reports';"
DbOperation(Qry)
End Sub
Public Function DbOperation(ByVal qry As String, Optional ByVal type As DatabaseQueryReturnType = DatabaseQueryReturnType.NonQuery) As Object
Dim objDB As New Object
Dim dt As DataTable
Dim da As SqlDataAdapter
OpenConnection()
Dim cmd As New SqlCommand(qry)
cmd.Connection = con
If type = DatabaseQueryReturnType.DataTable Then
dt = New DataTable
da = New SqlDataAdapter(cmd)
Try
da.Fill(dt)
Catch ex As Exception
MessageBox.Show("Error retrieving data from database: " & ex.Message.ToString)
End Try
objDB = dt
dt.Dispose()
CloseConnection()
Return objDB
ElseIf type = DatabaseQueryReturnType.NonQuery Then
Try
cmd.ExecuteNonQuery()
Catch ex As Exception
MessageBox.Show("Error executing nonquery: " & ex.Message.ToString)
End Try
objDB = Nothing
CloseConnection()
Return objDB
End If
Return objDB
End Function
When I try running the program, it gives me “The calling thread must be STA, because many UI components require this.” on the line that calls my function. After a bit of research, I learned that you have to call SetApartmentState() before the thread is started, but this doesn’t make sense since the recommended code seems to be:
Thread.CurrentThread.SetApartmentState(ApartmentState.STA)
After adding this line, I then get “Failed to set the specified COM apartment state” on that line of code.
I have also tried commenting out the messagebox lines in the function in case it was trying to open the messageboxes on the UI thread, but it had no impact and I still got the same errors.
What am I missing in my code to get the function to run on the BGW thread?

Need a fix for my deadlocking issue with my parallel.foreach

When running my code I seem to encounter deadlocks while trying to update a GUI element from within one of the parallel tasks.
I've tried surrounding the Output function with "Synclock me" to try to ensure that only one task is trying to update the control at a time.
Private Sub RunParallel(records as list(of DataRecord), ou as String)
Dim ParallelOptions As New ParallelOptions
ParallelOptions.MaxDegreeOfParallelism = 10
Parallel.ForEach(records, ParallelOptions, Sub(myrecord)
ProcessRecord(myrecord, ou)
End Sub)
Output("Done...." & vbCrLf)
End Sub
Private Sub ProcessRecord(ByVal record As DataRecord, ByVal ou As String)
'Output($"BromcomID = {record("ID")}, Forename = {record("Forename")}{vbCrLf}")
Dim ud As New UserDetails With {
.EmployeeID = record("ID"),
.SamAccountName = record("SamAccountName"),
.GivenName = record("Forename"),
.Surname = record("Surname")
}
If Not CreateUser(ou, ud) Then
'Threading.Thread.Sleep(2000)
' Output($"Error creating {ud.EmployeeID}{vbCrLf}")
End If
End Sub
Private Sub Output(ByVal s As String)
SyncLock Me
If Me.InvokeRequired Then
Invoke(Sub()
Outbox.AppendText(s)
Outbox.SelectionStart = Len(Outbox.Text)
Outbox.ScrollToCaret()
Outbox.Select()
End Sub)
Else
Outbox.AppendText(s)
Outbox.SelectionStart = Len(Outbox.Text)
Outbox.ScrollToCaret()
Outbox.Select()
End If
End SyncLock
End Sub
The code as supplied seems to run, but if I uncomment out the Output calls in the ProcessRecord() function, it hangs and never gets exits the Parallel.foreach
--- Update
After playing around with suggestions and comments on here I still can't get it to work correctly.
If I take out all of the output from ProcessRecord it seems to work correctly. However with the following code, it now seems to run each ProcessRecord sequentially (not 10 at a time as I intended), and then hangs after the last one.
Output("Dispatching" & vbCrLf)
Dim ParallelOptions As New ParallelOptions With {
.MaxDegreeOfParallelism = 10
}
Parallel.ForEach(recordList, ParallelOptions, Sub(myrecord)
ProcessRecord(myrecord, ou)
End Sub)
'For Each myrecord As DataRecord In recordList
' Task.Factory.StartNew(Sub() ProcessRecord(myrecord, ou))
'Next
Output("Done...." & vbCrLf)
End Sub
Private Sub ProcessRecord(ByVal record As DataRecord, ByVal ou As String)
Dim ud As New UserDetails With {
.EmployeeID = record("ID"),
.SamAccountName = record("SamAccountName"),
.GivenName = record("Forename"),
.Surname = record("Surname"),
.DisplayName = $"{record("Forename")} {record("Surname")} (Student)"}
If Not CreateUser(ou, ud) Then
' Output($"Error creating {ud.EmployeeID}{vbCrLf}")
End If
Output($"BromcomID = {record("ID")}, Forename = {record("Forename")}{vbCrLf}")
End Sub
Private Sub Output(ByVal s As String)
If Me.InvokeRequired Then
Invoke(Sub()
Output(s)
End Sub)
Else
Outbox.AppendText(s)
Outbox.SelectionStart = Outbox.TextLength
Outbox.ScrollToCaret()
Outbox.Select()
Outbox.Refresh()
End If
End Sub
If I use the commented out Task.Factory code everything seems to work perfectly, except I cant control how many tasks at a time are launched, and I can't wait till all of them have finished, the for loop just launches all the tasks, and then carries on with the Output("Done....) line.
The synclock statements didn't seem to affect anything either way.
Give this a try
Private Sub Output(ByVal s As String)
If Me.InvokeRequired Then
Me.Invoke(Sub() Output(s))
'Me.BeginInvoke(Sub() Output(s))
Else
Outbox.AppendText(s)
Outbox.SelectionStart = Outbox.TextLength
Outbox.ScrollToCaret()
Outbox.Select()
Outbox.Refresh()
End If
End Sub
There may be an issue if you have events tied to Outbox, like text changed. Tested Output method with
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim nums As New List(Of Integer)
For x As Integer = 1 To 500
nums.Add(x)
Next
'because it is in a button, run from a task
Dim t As Task
t = Task.Run(Sub()
Parallel.ForEach(nums, Sub(num)
Output(num.ToString & Environment.NewLine)
End Sub)
End Sub)
End Sub
If you want to go ahead with using a Task-based approach, then you certainly can control how many are launched at a time, and wait for all of them to finish. It requires some additional code for the manual management. This is discussed in some detail in Microsoft documentation: https://learn.microsoft.com/en-us/dotnet/standard/asynchronous-programming-patterns/consuming-the-task-based-asynchronous-pattern
It's not necessarily a bad thing to initiate all of the tasks immediately, then you'll be leaving it to the thread pool to take care of how many to run at a time.
If you want greater control, you can use the "throttling" design from the link. In your "pending" queue, store delegates/lambdas that will themselves kick off Task.Run. Then, as you dequeue from the "pending" queue into the "active" list, you can Invoke on the delegate/lambda to get the Task and Await Task.WhenAny on the "active" list.
One potential benefit of doing things this way is that the work in each top-level Task can be split between UI work running on the UI thread and processor-limited work running on the thread pool.
(I'm not suggesting that this is necessarily the best option for you, just trying to expand on what you should be looking at doing if you really want to pursue using Task instead of Parallel.)

How to update progressbar when retrieve data from database to datagridview VB.NET

i have VB.net code to retrieve data from SQL server (stored proc) to Datagridview (dgMC).
everything worked fine, but the progressbar1 not update. i'd like to progressbar1 shows update in percentage for user knows the status of data retrieve. the data is around 1000K.
Friend Delegate Sub SetDataSourceDelegate(table As DataTable)
Private Sub setDataSource(table As DataTable)
' Invoke method if required:
If Me.InvokeRequired Then
Me.Invoke(New SetDataSourceDelegate(AddressOf setDataSource), table)
Else
dgMC.DataSource = table
ProgressBar1.Visible = False
End If
End Sub
Private Sub loadTable()
Dim cnn As SqlConnection = GetConnection()
Dim cmdSearch As New SqlCommand("MC_Display", cnn)
cmdSearch.CommandType = CommandType.StoredProcedure
Try
cnn.Open()
Dim readerSearch = cmdSearch.ExecuteReader
If readerSearch.HasRows Then
Dim dt = New DataTable()
dt.Load(readerSearch)
setDataSource(dt)
Else
Me.Cursor = Cursors.Default
MsgBox("No Data Found.", MsgBoxStyle.Exclamation)
dgMC.DataSource = Nothing
End If
readerSearch.Close()
Catch ex As Exception
Throw ex
Finally
cnn.Close()
End Try
End Sub
Private Sub btnGoMC_Click(sender As Object, e As EventArgs) Handles btnGoMC.Click
ProgressBar1.Visible = True
ProgressBar1.Style = ProgressBarStyle.Marquee
Dim thread As New System.Threading.Thread(New System.Threading.ThreadStart(AddressOf loadTable))
thread.Start()
End Sub
To know exactly the number of data retrieve, you have to create a query with count(*) of your data.
Then, when you are retrieving data, you have to know which row is being retrieve, because you have to calculate the percentage.
And finally, you refresh your progressBar :
Dim percentage As Double = (currentRow / totalRows) * 100
ProgressBar.Value = Int32.Parse(Math.Truncate(percentage).ToString())
I hope it helps you
Since you're not performing anything that is quantifiable (you're issuing a sql query against a database, blocking processing for that thread until the database gives you the data you requested) you're not going to be able to update the progress bar with anything meaningful. In cases like this a Marquee scroll is usually sufficient. Also, you have the progress bar on the UI thread so it can be responsive and kick off a new thread to keep the UI thread from being blocked. At this moment you can't access the progress bar on the UI thread easily.
Some other thoughts though... If you have access to the Task Parallel Library, I would advise using that instead of creating a raw Thread and starting the process execution. You can utilize a type in the TPL called Task, which is an abstraction of Thread and takes care of some details you probably don't need to concern yourself with in this particular application/scenario. It also yields powerful asynchronous programming through the Async/Await paradigm in .NET 4.5. There is an excellent blog series by a guy named Stephen Cleary who has great expertise on this paradigm: Stephen Cleary Async/Await
A brief Example:
Private Sub btnGoMC_Click(sender As Object, e As EventArgs) Handles btnGoMC.Click
ProgressBar1.Visible = True
ProgressBar1.Style = ProgressBarStyle.Marquee
Dim thread As New System.Threading.Thread(New System.Threading.ThreadStart(AddressOf loadTable))
thread.Start()
End Sub
Can become:
`Private Async Sub btnGoMC_Click(sender As Object, e As EventArgs) Handles btnGoMC.Click
ProgressBar1.Visible = True
ProgressBar1.Style = ProgressBarStyle.Marquee
Await Task.Run(Sub() loadTable)
End Sub`

Constantly losing TCPClient connection from application

I have an application which connects to a Zebra QLn420 label printer through a TCPClient connection to the printer's static IP address and port number.
We have been having some issues with the software "stalling" and skipping over some of the labels, and after investigating (by adding some logging capabilities to the software to write out to a text file each time it checks the connection and actually reconnects) I have noticed that the connection does not seem to stay open for more than one second (not a solid number, it can vary from less than a second to 5 seconds or more from my tests).
While testing at my desk, this is almost a non-issue because the signal strengths are so strong that it reconnects instantly, but out in the warehouse where this is actually used, in some areas the signal isn't so strong. The function which handles the connection will try to reconnect for up to 5 seconds twice, then display an error message and ask if they want to retry connecting.
I'm curious if anyone has experienced anything like this, and what can be done to resolve it. I understand that the software will lose connection with the printer occasionally, especially in the weaker signal areas. But I would like to find a way to keep the connection open until either I close it or the signal is too weak to keep it going, as I feel this has a lot to do with the "missing" labels and definitely impacts the speed of the software.
Below is my connection code (a combination of my own code and some snippets found online):
Public Class LabelPrinter
Private strIP as String = "xxx.xxx.xxx.xxx"
Private intPort As Integer = "6101"
Private client As System.Net.Sockets.TcpClient
Private blnConnected As Boolean = False
Private blnValidIP As Boolean = False
Private intTimeoutLength As Integer = 5000
Private TimeoutTime As New Timers.Timer
Private clntSockParams As ClientSocketParameters
Private Structure ClientSocketParameters
Public addrs As String
Public prt As Integer
End Structure
Public Sub New(Picker As Picker, newIP As String, newPort As Integer, Optional PrinterTimeout As Integer = 5000)
If newIP <> "" Then strIP = newIP
If newPort <> 0 Then intPort = newPort
intTimeoutLength = PrinterTimeout
If newIP = "" Or newPort = 0 Then Exit Sub
Dim ipAddress As System.Net.IPAddress = Nothing
If Not System.Net.IPAddress.TryParse(strIP, ipAddress) Then Exit Sub
End Sub
Public Function Connect() As Boolean
Try
'connect to printer via TcpClient, need ip address and port number
'connects without thread, hangs program for 10-20 seconds if printer is not turned on, replaced with code below to thread the connection and set timeout
'If client Is Nothing OrElse Not client.Connected Then
' client = New System.Net.Sockets.TcpClient
' client.Connect(strIP, intPort)
'End If
'RRB 02/10/15 - added for loop to try to connect twice each attempt instead of only once
For i As Integer = 1 To 2
If client Is Nothing OrElse Not client.Connected Then
'uses ClientSocketParameters structure to pass to recursive function ConnectionReturned()
clntSockParams = New ClientSocketParameters
clntSockParams.addrs = strIP
clntSockParams.prt = intPort
'create client and call BeginConnect (attempts to connect on separate thread until TimeoutTime has elapsed)
client = New System.Net.Sockets.TcpClient
client.SendTimeout = intTimeoutLength
client.ReceiveTimeout = intTimeoutLength
'setup timer with timeout length and start, if timer goes past intTimeoutLength, the Timeout() function is called which closes everything and leaves client = Nothing
AddHandler TimeoutTime.Elapsed, AddressOf Timeout
TimeoutTime.Interval = intTimeoutLength
TimeoutTime.Start()
client.BeginConnect(strIP, intPort, New AsyncCallback(AddressOf ConnectionReturned), clntSockParams)
'keeps the program from doing anything else until BeginConnect either succeeds or fails (due to connect on separate thread)
Do While TimeoutTime.Enabled
System.Threading.Thread.Sleep(500)
Loop
End If
'if TimeoutTime is elapsed and client is Nothing, connection didn't happen, throw an error
If client Is Nothing Then
blnConnected = False
Else
blnConnected = True
Exit For
End If
Next
Catch ex As Exception
blnConnected = False
End Try
Return blnConnected
End Function
Private Sub ConnectionReturned(ByVal ar As System.IAsyncResult)
'this method is called from the client.BeginConnect line in Connect(), make sure timer is running
If TimeoutTime.Enabled Then
'ensure client is initialized
If client Is Nothing Then client = New System.Net.Sockets.TcpClient
'keep calling ConnectionReturned until client.Connected is true
If client.Connected Then
TimeoutTime.Stop()
Else
Dim actualParameters As ClientSocketParameters = DirectCast(ar.AsyncState, ClientSocketParameters)
client.BeginConnect(actualParameters.addrs, actualParameters.prt, New AsyncCallback(AddressOf ConnectionReturned), clntSockParams)
End If
End If
End Sub
Private Sub Timeout(ByVal sender As Object, ByVal e As EventArgs)
'this method is only called if TimeoutTime elapsed, which means no connection was made. close the client object if needed, set to Nothing, and stop TimeoutTime
If TimeoutTime.Enabled Then
Try
client.Close()
Catch ex As Exception
End Try
client = Nothing
TimeoutTime.Stop()
End If
End Sub
Public Sub Disconnect()
'need to make sure StreamWriter connection and TcpClient connection to printer are closed
Try
client.Close()
Catch ex As Exception
End Try
client = Nothing
blnConnected = False
End Sub
End Class
Please let me know if more information is needed. I'm also open to alternative connection types, so long as the static IP and port can be used (over wifi connection).
EDIT: I've modified the code in my program to use the System.Net.Socket class, and the connection seems to hold steady in my initial tests. What would cause this connection method to work where TCPClient doesn't seem to, in the same location?

How to avoid Thread.Abort() in this case?

I know that Thread.Abort() is not really safe, but I cannot imagine what it could cause in case below:
Private threadLoadList As Thread
Private Sub LoadList(ByVal argument As Integer)
Try
Refresh_Grid(GET_DATA(argument), argument) 'Fills grid with data loaded from database
Change_Cursor() 'Changes cursor to Cursor.Default
Catch ex As ThreadAbortException
'Do nothing
Catch ex As Exception
Error_log(ex.Message) ' Saves ex.Message in database
End Try
End Sub
Private Sub SomeValueChanged(sender As Object, e As EventArgs) Handles Control.ValueChanged
If Not threadLoadList Is Nothing Then
If threadLoadList.IsAlive Then
threadLoadList.Abort()
End If
End If
Cursor = Cursors.WaitCursor
threadLoadList = New Thread(AddressOf LoadList)
threadLoadList.Start(1)
End Sub
As you can see, user can change some value (ComboBox) and in result change content of the grid. Function GET_DATA() takes around 10 seconds, thus it is possible that user changes value of Combobox before grid is refreshed - that is why previously started thread is killed.
Is it dangerous? If yes, could you propose some other solution?
Ok, I understand ;). I try to avoid timeouts (in some cases query executes below 1 second) Is it better solution:
Private threadLoadList As Thread
Private Changed As Boolean = False
Private Sub LoadList(ByVal argument As Integer)
Try
Dim dt As DataTable = GET_DATA(argument)
'Enter Monitor
If Changed Then
Return
End IF
'Exit Monitor
Refresh_Grid(dt, argument) 'Fills grid with data loaded from database
Change_Cursor() 'Changes cursor to Cursor.Default
'Enter Monitor
Changed = False
'Exit Monitor
Catch ex As ThreadAbortException
'Do nothing
Catch ex As Exception
Error_log(ex.Message) ' Saves ex.Message in database
End Try
End Sub
Private Sub SomeValueChanged(sender As Object, e As EventArgs) Handles Control.ValueChanged
'Enter Monitor
Changed = True
'Exit Monitor
Cursor = Cursors.WaitCursor
Dim t As Thread = New Thread(AddressOf LoadList)
t.Start(1)
End Sub
First things first, unless there is extra code in Refresh_Grid that I am not seeing it looks like you are attempting to modify a UI element from a thread other than the UI thread. That is a big no-no. UI elements are designed to be accessed only from the UI thread so attempting to modify one on a worker thread will lead to unpredictable behavior.
In regards to your usage of Thread.Abort the answer is yes; it is dangerous. Well, it is dangerous in the sense that your application will throw an exception, corrupt data, crash, tear a hole in spacetime, etc. There are all kinds of failure modes that might be expected from aborting a thread. The best solution is to use a cooperative termination protocol instead of doing a hard abort. Refer to my answer here for a brief introduction on valid approaches.