Calling a Sub that Connects to DB from Thread - vb.net

Question

I'm writing a VB.net 2017 Windows Service that looks at SQL and depending on the number of rows, it will create multiple threads. These threads monitor folders and report back to a different table and log the data accordingly. This piece of code has been running for a few years and has been working very well but in the last few days, I've decided to switch it from a console application that runs on start up to a window service and this is my first time writing a windows service.
I've went through and got the code to work but testing was a major pain because I couldn't walk through the code. I made some changes and consolidated some of the duplicate sections. For example, I was writing 4 or 5 different sections of code to either write data to SQL or pull data from SQL. I consolidated these down to only 2 sub routines and the threads continuously use them. Depending on the situation, the program can have between 1-15 threads and when I start activating more threads, I started to run into an issue. I already had try statements in my code from the console app before I ported it and I just put these into a log table when creating the new program and it was complaining that I was trying to Open an "Open connection". Here is an example of one the routines that pulls data from SQL:
Public con As New SqlClient.SqlConnection
Public dsGeneral As New DataSet
Public dc1 As SqlClient.SqlCommand
Public pullGeneral As SqlClient.SqlDataAdapter
Public maxRowsGeneral As Integer
Public Sub PullGeneralSQL(ByVal SQL As String)
Try
If (con.State = ConnectionState.Closed) Then
con.Open()
End If
dsGeneral.Tables.Clear()
pullGeneral = New SqlClient.SqlDataAdapter(SQL, con)
pullGeneral.Fill(dsGeneral, "General")
maxRowsGeneral = dsGeneral.Tables("General").Rows.Count
Catch ex As Exception
Msg(ex.Message, "Error")
maxRowsGeneral = 0
End Try
con.Close()
End Sub
I'm also getting errors saying that the connection is already closed as well. I'm assuming that another thread has finished connecting and closed the connection while a thread was in the middle of a task.
My question is, what is the best way to handle this?

Answer 1

I have run into this issue before even though there is connection pooling eventually as you open and close and open and close connections the database ends up throwing errors like you are getting. In my case it was 600 threads over and over. Worked for a little while then threw the same types of errors.
The solution for me was to create my own connection pool.
Public Class Form1
Public dsGeneral As New DataSet
Public dc1 As SqlClient.SqlCommand
Public pullGeneral As SqlClient.SqlDataAdapter
Public maxRowsGeneral As Integer
Public Sub PullGeneralSQL(ByVal SQL As String)
'Get a connection from the list..Thread safe
'Your thread will wait here until there is a connection to grab.
'Since your max is 15 the 16++ threads will all wait their turn
Dim con As SqlClient.SqlConnection = ConnectionManager.WaitForConnection()
Try
dsGeneral.Tables.Clear()
pullGeneral = New SqlClient.SqlDataAdapter(SQL, con)
pullGeneral.Fill(dsGeneral, "General")
maxRowsGeneral = dsGeneral.Tables("General").Rows.Count
Catch ex As Exception
Msg(ex.Message, "Error")
maxRowsGeneral = 0
End Try
'Put the connection back into the list
'Allows another thread to start
ConnectionManager.ReleaseConnection(con)
End Sub
End Class
Public Class ConnectionManager
Public Shared poolLimit As Integer = 15 'Maximum number of connections to open.
Public Shared connectionString As String = "PUT YOUR CONNECTION STRING HERE"
'Since this is static it will create 15 connections and add them to the list when the service starts
Private Shared ConnectionList As New IThreadPool(Of SqlClient.SqlConnection)(Function()
Dim connection As New SqlClient.SqlConnection(connectionString)
connection.Open()
Return connection
End Function, poolLimit)
''' <summary>
''' Gets the pool count.
''' Purely for information to get the number of connections left in the list
''' </summary>
Public ReadOnly Property PoolCount() As Integer
Get
Return ConnectionList.PoolCount
End Get
End Property
''' <summary>
''' Waits until there is a free connection in the list
''' When there is a free connection grab it and hold it
''' </summary>
Public Shared Function WaitForConnection() As SqlClient.SqlConnection
Try
Return ConnectionList.GetObject()
Catch ex As Exception
'only during close
Throw ex
End Try
Return Nothing
End Function
''' <summary>
''' Releases the connection.
''' Put the connection back into the list.
''' </summary>
Public Shared Sub ReleaseConnection(connection As SqlClient.SqlConnection)
Try
ConnectionList.PutObject(connection)
Catch ex As Exception
'only during close
Throw ex
End Try
End Sub
''' <summary>
''' Disposes this instance.
''' Make sure to dispose when the service shuts down or your connections will stay active.
''' </summary>
Public Shared Sub Dispose()
ConnectionList.Dispose()
End Sub
End Class
Public Class IThreadPool(Of T)
Private connections As System.Collections.Concurrent.BlockingCollection(Of T)
Private objectGenerator As Func(Of T)
Public Sub New(objectGenerator As Func(Of T), boundedCapacity As Integer)
If objectGenerator Is Nothing Then
Throw New ArgumentNullException("objectGenerator")
End If
connections = New System.Collections.Concurrent.BlockingCollection(Of T)(New System.Collections.Concurrent.ConcurrentBag(Of T)(), boundedCapacity)
Me.objectGenerator = objectGenerator
Task.Factory.StartNew(Function()
If connections.Count < boundedCapacity Then
Parallel.[For](0, boundedCapacity, Function(i)
Try
If connections.Count < boundedCapacity Then
connections.Add(Me.objectGenerator())
End If
Catch ex As Exception
'only error during close
End Try
End Function)
Try
While connections.Count < boundedCapacity
connections.Add(Me.objectGenerator())
End While
Catch ex As Exception
'only error during close
End Try
End If
End Function)
End Sub
Public ReadOnly Property PoolCount() As Integer
Get
Return If(connections IsNot Nothing, connections.Count, 0)
End Get
End Property
Public Function GetObject() As T
Return connections.Take()
End Function
Public Sub PutObject(item As T)
connections.Add(item)
End Sub
Public Sub Dispose()
connections.Dispose()
End Sub
End Class