I have an app that reads a csv file with an image name, calls a sub that opens the image, scans to find a yellow pixel in a shape, and resolves the upper left corner, annotates the csv data and writes the record. Managed Heap and Native heap memory is stable, but process memory grows 350MB per image. For now I have my analysts chop the csv file into 25 image sets. But this is risky with 20-ish analysts.
I dispose the images, tried GCCollect(), large heapCompaction - nothing seems to help. I have read most every post and nothing seems to ring. I have included code - and tried to strip out displays and junk but left some of the scan comparisons.
I am running .net 4.6.1, Win 10/64, 16GB memory
Sub Process_CSV_Detail()
. . . read csv file sequentially, set up file names, display for user, than call working process for each image - the memory leak seems to occur as each new image is accessed
Call BackgroundProcess2()
End Sub
Public Sub BackgroundProcess2()
GreenTest = 245
RedTest = 245
BlueTest = 70
Try
loadedImage = Image.FromFile(InputImageName)
Catch ex As Exception
. . . .Never gets here but some code
Finally
End Try
HeightVal = loadedImage.Height
WidthVal = loadedImage.Width
Dim MasterImage As New Bitmap(WidthVal, HeightVal, FormatVal)
MasterImage = loadedImage
. . . Now looking for that pesky pixel
For ycounter = 1 To HeightVal - 1 Step PixelStride
For xcounter = 1 To WidthVal - 1 Step PixelStride
Dim PixelColor As Color = MasterImage.GetPixel(xcounter, ycounter)
PixelIsYellow = False
If PixelColor.R > RedTest Then
If PixelColor.G > GreenTest Then
If PixelColor.B < BlueTest Then
PixelIsYellow = True
YellowPixelCount = YellowPixelCount + 1
MasterImage.SetPixel(xcounter, ycounter, Color.FromArgb(&HFFFFFF00))
xPixelIsYellow = True
yPixelIsYellow = True
End If
End If
End If
If PixelIsYellow = True Then
'Now find the upper left corner
LeftXLoc = xcounter
LeftYLoc = ycounter
'Move to left until no more yellow, then back 1 step to
'locate left pixel location
Try
For xtestcounter = LeftXLoc To 1 Step -1
Dim PixelColorx As Color = MasterImage.GetPixel(xtestcounter, LeftYLoc)
If PixelColorx.R < RedTest Then
xPixelIsYellow = False
Exit Try
End If
If QA_Mode = False Then
If PixelColorx.G < GreenTest Then
xPixelIsYellow = False
Exit Try
End If
End If
If QA_Mode = True Then
If PixelColorx.G < GreenTest Then
xPixelIsYellow = False
Exit Try
End If
End If
If PixelColorx.B > 70 Then
xPixelIsYellow = False
Exit Try
End If
Next
Catch ex As Exception
Finally
End Try
LeftXLoc = xtestcounter + 1
'Move up until no more yellow, then back 1 step to locate left pixel location
Try
For ytestcounter = LeftYLoc To 1 Step -1
Dim PixelColory As Color = MasterImage.GetPixel(LeftXLoc, ytestcounter)
If PixelColory.R < RedTest Then
yPixelIsYellow = False
Exit Try
End If
If PixelColory.G < GreenTest Then
yPixelIsYellow = False
Exit Try
End If
If PixelColory.B > BlueTest Then
xPixelIsYellow = False
Exit Try
End If
Next
Catch ex As Exception
MsgBox("Error in locating upper left pixel")
Finally
End Try
LeftYLoc = ytestcounter + 1
OutputLine = CurrentDataLine & "," & xcounter & "," & ycounter & "," & LeftXLoc & "," & LeftYLoc
PrintLine(TargetFileNumber, OutputLine)
End If
Next
Next
loadedImage.Dispose()
MasterImage.Dispose()
' - I have tried these but no effect
'GC.Collect()
'Runtime.GCSettings.LargeObjectHeapCompactionMode = Runtime.GCLargeObjectHeapCompactionMode.CompactOnce
'Finalize()
End Sub
I expect that someone will have a silver bullet that allows process memory to be stable - Ive tried ANTS but no joy.
These two lines are (at least part of) the problem:
Dim MasterImage As New Bitmap(WidthVal, HeightVal, FormatVal)
MasterImage = loadedImage
You create a new bitmap with the specified dimensions and pixel format, then you immediately replace the MasterImage variable's reference to the new bitmap with that of your loadedImage. Now the new bitmap doesn't have any references, is not disposed and therefore lives on in memory until you close the process. And instead, MasterImage now refers to the exact same bitmap as loadedImage.
Thus, when your code disposes of the bitmaps, it is actually trying to dispose of the same bitmap twice:
loadedImage.Dispose()
MasterImage.Dispose() 'This is the exact same bitmap as loadedImage, which is already disposed of.
Image data in GDI+ is unmanaged memory, which is why the managed memory graph remains stable. Unmanaged memory is simply speaking any piece of memory that is not managed by a Garbage Collectior (GC), which is why calling any of its methods doesn't help. It must be freed manually by the programmer (in this case by calling Dispose()).
The solution is to just not create that new bitmap at all since you never actually use it. Remove the MasterImage variable altogether and operate on loadedImage directly.
I have an attribute tab delimited text file that I want to apply to multiple drawings. In order for AutoCAD to NOT pop up and say "One or more blocks could not be found, do you want to select the data interactively?" , I have to use the HANDLE property of the block. On a given drawing, if I use ATTOUT to see the Handle of my block, I get a value such as '8B3F. Using ATTIN with that Handle works. Applying this to multiple drawings that have different handles, I have to get the handle for each block if each drawing. Here is my code - writing the handle to an excel doc.
xlbook = xlapp.Workbooks.Open(attInText,, False)
xlsheet = xlbook.Worksheets(dwgName)
Dim Handle As String = ""
'get the handle to the CHS11x17TB title block
For Each blk As AutoCAD.AcadBlock In cadDOC.Blocks
If blk.Name.ToUpper = "CHS11X17TB" Then
Handle = blk.Handle
xlsheet.Cells(2, "A").value = Handle
Exit For
End If
Next
Now, the problem is that the Handle is NOT the same as the one generated using ATTOUT - I'll get something like '75B0 using the code. Why do you think ATTOUT gives me a different handle than looping through the blocks of the drawing? I would note that my block is in paperspace, if that makes any difference. If that question cannot be answered, I'm interested in any alternative solutions for getting the handle to my block.
It looks like you're confusing block definition (Block) contained in the block table (Blocks) and block reference (BlockReference) inserted in the ModelSpace or PaperSpace.
Here's a not tested snippet which serac for a block reference in the model space (you can replace ModelSpace with PaperSpace to search the active paper space.
xlbook = xlapp.Workbooks.Open(attInText,, False)
xlsheet = xlbook.Worksheets(dwgName)
Dim Handle As String = ""
'get the handle to the CHS11x17TB title block
For Each obj As AutoCAD.AcadObject In cadDOC.ModelSpace
If obj.ObjectName = "AcDbBlockReference" Then
If obj.EffectiveName.ToUpper = "CHS11X17TB" Then
Handle = obj.Handle
xlsheet.Cells(2, "A").value = Handle
Exit For
End If
End If
Next
Here's what I did to make it work. The block reference I wanted was in the paperspace. Note that EntityType 7 is an AcadBlockReference.
Dim Handle As String = ""
Dim count As Integer
count = cadDOC.PaperSpace.Count
Dim newObjs(count) As AutoCAD.AcadEntity
Dim index As Integer
For index = 0 To count - 1
newObjs(index) = cadDOC.PaperSpace.Item(index)
Next
For i = 0 To count - 1
Try
If newObjs(i).EntityType = 7 Then
Dim blk As AutoCAD.AcadBlockReference = newObjs(i)
If blk.Name.ToUpper = "CHS11X17TB" Then
Handle = "'" & blk.Handle
End If
End If
Catch ex As Exception
End Try
Next
I am working on a VB.net application where I have a very large text file. It is basically a large database of error codes with descriptions of how to clear the code after it. What I would like to do, is on the click of a button, search the text file for the specific code and display all text for just that error code into a text box. I have tried many different ways, but am unable to get it to work properly. I went through the entire text file and added a "|" to the beginning of each fault code so that I could specify where the code starts at.
Here is an example of a couple fault codes:
|ACAL-000 Fail to run DETECT Motn Cause: The AccuCal2 Motion failed to
nm. The AccuCal2 motion cannot be started. Remedy: Clear all the
errors before executing AccuCal2. |ACAL-001 Robot is not ready.
Cause: The robot is not ready. The system cannot issue motion
because it is in an error state. Remedy: Clear all faults, then retry
the operation.
If I search for "ACAL-000", I want it to show everything from the | before ACAL-000 to the bar before ACAL-001.
I would post the code that I have written, but I have tried so many different versions that I don't really know which one to post.
Any help you can provide would be greatly appreciated.
EDIT
Here is my current code after some editing and implementation of what has been recommended. Please see the comments below for more information on how I got to this point. A quick note, I am currently just using "|ACAL-000" for a test search. When this is complete, I have some other (already working) code that will put together a code from a couple of drop down lists.
Function ReadEmbeddedTextFileResource(embeddedResourceName As String) As String
Using stream As Stream = Reflection.Assembly.GetExecutingAssembly().GetManifestResourceStream(embeddedResourceName)
If stream Is Nothing Then
Throw New FileNotFoundException("The resource '" & embeddedResourceName & "' was not found.")
End If
Using reader As StreamReader = New StreamReader(stream, True)
Return reader.ReadToEnd()
End Using
End Using
End Function
Function FindTextBetweenBars(bodyOfTextToSearch As String, textToLookFor As String) As String
Dim i As Integer = bodyOfTextToSearch.IndexOf(textToLookFor)
If i < 0 Then Return Nothing
Dim j As Integer = bodyOfTextToSearch.LastIndexOf("|", i)
If j < 0 Then j = 0
Dim k As Integer = bodyOfTextToSearch.IndexOf("|", i + Len(textToLookFor))
If k < 0 Then k = Len(bodyOfTextToSearch)
Return bodyOfTextToSearch.Substring(j + 1, k - j - 1)
End Function
Private Sub btnShowTroubleshooting_Click(sender As Object, e As EventArgs) Handles btnShowTroubleshooting.Click
Dim allErrorText As String = ReadEmbeddedTextFileResource(My.Resources.FanucCodes)
Dim errorMessage As String = FindTextBetweenBars(allErrorText, "|ACAL-000")
If errorMessage Is Nothing Then errorMessage = "Error code Not found!"
RichTextBoxFanucFaults.Text = errorMessage
End Sub
Here is a function that should do what you want:
Function FindTextBetweenBars(bodyOfTextToSearch As String, textToLookFor As String) As String
Dim i As Integer = bodyOfTextToSearch.IndexOf(textToLookFor)
If i < 0 Then Return Nothing
Dim j As Integer = bodyOfTextToSearch.LastIndexOf("|", i)
Dim k As Integer = bodyOfTextToSearch.IndexOf("|", i + Len(textToLookFor))
If k < 0 Then k = Len(bodyOfTextToSearch)
Return bodyOfTextToSearch.Substring(j + 1, k - j - 1)
End Function
In your button click event handler you can call the function like this:
Dim errorMessage as String = FindTextBetweenBars(My.Resources.FanucCodes, txtErrorCodeToLookFor.Text)
If errorMessage Is Nothing Then errorMessage = "Error code not found!"
txtErrorMessage.Text = errorMessage
where txtErrorMessage is the output textbox to display the error message result,
My.Resources.FanucCodes is your large string resource containing all the error descriptions (with | separators), and txtErrorCodeToLookFor is a textbox that accepts the error code input from the user.
Experts,
This is what happens when you can't find a solution anywhere on the Interwebs, and you just have to hack at it until you get it looking good [enough].
I have a situation where I need to parse objects coming in at high speed, but the parsing takes a relatively long time-far slower than they come in... I can tell how many cores are on a box pretty easily, and then I can carve up a team of worker threads (Tasks) to all divide & conquer! But the problem with ANY multi-threaded application is "How Many Threads?"
There is no hard & fast answer, so don't bother looking. My approach is to make a flexible approach that my main thread can monitor to see if throughput (aggregate work completed in X amount of time) is at maximum for the machine it happens to be running on. Also, that same machine may vary in load, RAM available, etc. over time, so you can't just set it & forget it...
I'm just asking to answer my own question, which SO encourages.
If the parsers are expected to be busy all the time (or nearly so), it makes no sense to have more parser threads than CPU threads that can work on them. Having 10 parser threads when you have only 4 cores is pointless because there will be overhead in thread context switches. So allocate 3 or 4 worker threads (consumers) that service the queue.
See https://stackoverflow.com/a/2670568/56778 to determine the number of logical processors that are on your system. It makes no sense to have more worker threads than that. Whether it makes sense to use your complicated scheme of dynamically allocating workers is ... a matter of opinion. I'd be more inclined to have a timer that examines the queue state (number of items) once a minute, and have it allocate or deallocate worker threads appropriately, adding or deleting one worker per minute in order to avoid overshooting the mark. That would probably make very good use of avaliable resources.
Removing a worker would be very easy. If you create an AutoResetEvent that each thread checks each time through its loop, then the first thread that sees it can exit. For example:
private AutoResetEvent _killAThread = new AutoResetEvent(false);
// in each thread
while (!_killAThread.Wait(0) && !cancelToken.IsCancellationRequested)
{
// thread waits for an item from the queue and processes it.
}
Granted, this could cause you to have too many threads waiting if the queue is empty for a long period, but I get the sense that you don't expect that condition to occur very often (if at all).
And, of course, adding a new consumer is easy enough. Your timer tick handler compares the existing queue size against some threshold and spins up a new thread if required. Or, if there are too many threads it calls _killATheread.Set(), and the next thread to finish its processing will check the event, see that it's set, and exit.
[Note: Line numbers refer to those inserted with Crayon for Wordpress at original blog post here. At that link, you can also download a 7z of the entire VS 2012 solution.]
So this is the idea:
So let's see how I solved the issue of the blocking collection and then dishing our work to the Parser Tasks... I used a very generic, and easily re-usable approach of Producer and Consumer. In fact, in my example below, you can even adjust the amount of Producers as well. You can adjust my code below to more closely approximate your work, and then adjust the # of threads/tasks/Consumers to see how well parallelism can work for you.
First, the usual suspects in Imports..
Imports System.Threading
Imports System.Threading.Tasks
Imports System.Collections.Concurrent
Module modStartHere
'
' Producer
'
Dim itemsToProduce = 10
Dim sleepProducer = 10 ' in milliseconds
Dim producerStartID = 1
Dim producersNumToStart = 1
Dim ProducerCTSs As New ConcurrentBag(Of CancellationTokenSource)
Dim moreItemsToAdd As Boolean = True
'
' Consumer
'
Dim sleepConsumer = 1000 ' in milliseconds
Dim consumerStartID = 100
Dim consumersNumToStart = 3
Dim ConsumerCTSs As New ConcurrentBag(Of CancellationTokenSource)
The Producer(s) are initially set up with the above. While the itemsToProduce doesn't change during the program, the number of Producers & Consumers will. This is very rough as a draft, and it will no doubt be streamlined in your own code at some point, but this demonstrates how to solve this problem very well.
I used "IDs" to show in the output which thread was doing what. They only thing needed in production is the list of CTS instances:
'
' the multi-thread-safe queue that is produced to and consumed from
'
Dim bc As New BlockingCollection(Of Integer)
'
' this # will be what is actually produced & consumed (1, 2, 3, ...)
'
Dim itemId As Integer = 0
'
The main machine here is that one little line:
Dim bc As New BlockingCollection(Of Integer)
Microsoft says:
BlockingCollection Overview .NET Framework 4.5
BlockingCollection(Of T) is a thread-safe collection class that
provides the following features:
An implementation of the Producer-Consumer pattern.
Concurrent adding and taking of items from multiple threads.
Optional maximum capacity.
Insertion and removal operations that block when collection is empty or full.
Insertion and removal "try" operations that do not block or that block up to a specified period of time.
Encapsulates any collection type that implements IProducerConsumerCollection(Of T)
Cancellation with cancellation tokens.
Two kinds of enumeration with foreach (For Each in Visual Basic):
-Read-only enumeration.
-Enumeration that removes items as they are enumerated.
itemId is just a variable that holds the fake payload. Producers will increment it by one to simulate a different object instance, or unit of work. You just change the type the BlockingCollection holds...
Now I'm not doing this in a FIFO way (which I will in production), but you can either do that, or even a FILO, as per Microsoft:
When you create a BlockingCollection(Of T) object, you can specify not
only the bounded capacity but also the type of collection to use. For
example, you could specify a ConcurrentQueue(Of T) object for first
in, first out (FIFO) behavior, or a ConcurrentStack(Of T) object for
last in, first out (LIFO) behavior.
Now THAT is useful! Here in this demo, I did it all willy-nilly... But like I said, for my specific need, I need FIFO, as in the diagram at the top...
Later, you will see the functions and subroutines, but the real magic here is in 2 collections-one for Producers and one for Consumers:
Dim ProducerCTSs As New ConcurrentBag(Of CancellationTokenSource)
Dim ConsumerCTSs As New ConcurrentBag(Of CancellationTokenSource)
The Magic: As each Task(thread) is either created or closed, the corresponding CancellationTokenSource is either added or removed from the appropriate collection above.
Seriously, that is it! :)
Next in the code, the initial producers and consumers are created:
'===============================
'
' start demo
'
Sub Main()
'
'===============================
'
' initial state:
'
' start our producer(s)
'
For ps As Integer = producerStartID To producerStartID + producersNumToStart - 1
CreateTask(ps, "Producer")
Next
'
' start our consumer(s)
'
For cs As Integer = consumerStartID To consumerStartID + consumersNumToStart - 1
CreateTask(cs, "Consumer")
Next
'
'=========================================
Aside from a few Thread.Sleep() calls, all the next part does is add or remove producer and consumer Tasks(threads). You can vary the initial values at the top to put it through the paces.
To create a Task... - CreateTask(, <"Producer" or "Consumer">)
To remove a Task, you (in one line) both get a random CTS, and then .Cancel() it:
GetRandomCTS(ProducerCTSs).Cancel()
GetRandomCTS(ConsumerCTSs).Cancel()
GetRandomCTS() takes the collection of CTS instances, picks one at random, then calls Cancel() on it.
'
Thread.Sleep(2000)
'
' create a producer
'
Console.WriteLine("creating producer 555...")
CreateTask(555, "Producer")
Thread.Sleep(1000)
'
' cancel a consumer
'
Console.WriteLine("cancelling random consumer...")
GetRandomCTS(ConsumerCTSs).Cancel()
Thread.Sleep(2000)
'
' cancel a consumer
'
Console.WriteLine("cancelling random consumer...")
GetRandomCTS(ConsumerCTSs).Cancel()
Thread.Sleep(1000)
'
' create a consumer
'
Console.WriteLine("creating consumer 222...")
CreateTask(222, "consumer")
Thread.Sleep(1000)
'
' cancel a producer
'
Console.WriteLine("cancelling random producer...")
GetRandomCTS(ProducerCTSs).Cancel()
Thread.Sleep(1000)
'
' cancel a consumer
'
Console.WriteLine("cancelling random consumer...")
GetRandomCTS(ConsumerCTSs).Cancel()
'
'==========================================
'
Console.ReadLine()
End Sub
And that is it!
Now for the fun parts:
#Region "Utilites"
''' <summary>
''' Retrieves a random cancellation token source from the given list of current threads...
''' Works for either producer or consumer
''' </summary>
''' <param name="ctsBag">ConcurrentBag(Of CancellationTokenSource)</param>
''' <returns>CancellationTokenSource</returns>
''' <remarks></remarks>
Function GetRandomCTS(ctsBag As ConcurrentBag(Of CancellationTokenSource)) As CancellationTokenSource
Dim cts As CancellationTokenSource = Nothing
Dim rndNum As Random = Nothing
Dim rndIndex As Integer = Nothing
Try
If ctsBag.Count = 1 Then
Console.WriteLine("There are no threads to cancel!")
Else
rndNum = New Random(12345)
rndIndex = rndNum.Next(0, ctsBag.Count - 1) ' because ElementAt() is zero-based index
cts = ctsBag.ElementAt(rndIndex)
End If
Catch ex As Exception
Console.WriteLine("GetRandomCTS() Exception: " & ex.StackTrace)
End Try
Return cts
End Function
Line 7: This is what we'll be returning, a CancellationTokenSource
Line 16: ctsBag.ElementAt() allows us to pull out a specific CTS instance by number.
Below, the CreateTask takes an argument for the # you want it to display when running (just for demo, to see which thread is doing what), and a string to tell it whether you want a new Producer of Consumer. Sure, I could have made it more complex, but this is just a rough draft. :)
Private Function CreateTask(taskId As Integer, taskType As String) As CancellationTokenSource
Dim t As Task = Nothing
Dim cts As New CancellationTokenSource()
Dim token As CancellationToken = cts.Token
Try
If taskType.ToLower = "producer" Then
t = Task.Factory.StartNew(Sub() Producer(taskId, token), token, TaskCreationOptions.LongRunning)
ProducerCTSs.Add(cts)
ElseIf taskType.ToLower = "consumer" Then
t = Task.Factory.StartNew(Sub() Consumer(taskId, token), token, TaskCreationOptions.LongRunning)
ConsumerCTSs.Add(cts)
Else
End If
Console.WriteLine("{0} Task {1} ({2}) running!", taskType, taskId.ToString("000"), t.Id)
Catch ex As Exception
Console.WriteLine("Task {0} CreateTask({1}) Exception: ", taskId.ToString("000"), taskType & ex.StackTrace)
End Try
Return cts
End Function
#End Region
Line 7 & 10: These call the Producer() or Consumer() classes below, passing them the CancellationTokenSource needed to let them be able to elegantly be cancelled while running without corrupting any data.
t = Task.Factory.StartNew(Sub() Producer(taskId, token), token, TaskCreationOptions.LongRunning)
Did you notice TaskCreationOptions.LongRunning? That is good in my case, and it improves performance by telling the program to not worry so much about wating what happens for cancellations too closely.
So what does a Producer() class look like?
#Region "Producer(s)"
Public Sub Producer(ByVal taskNum As Integer, ByVal ct As CancellationToken)
' Was cancellation already requested?
If ct.IsCancellationRequested = True Then
Console.WriteLine("Producer Task {0} was cancelled before Producer thread created!", taskNum.ToString("000"))
ct.ThrowIfCancellationRequested()
End If
'
'Dim r As Random = New Random(123)
Dim sw As New Stopwatch
Dim numAdded As Integer = 0
sw.Start()
While moreItemsToAdd = True
' Dim itemIn As Integer = r.Next(1, 1000)
itemId += 1 ' the payload
Try
bc.Add(itemId)
Console.WriteLine("--> " & taskNum.ToString("000") & " --> [+1 => Q has: " & bc.Count & "] added: " & itemId)
numAdded += 1
If ct.IsCancellationRequested Then
Console.WriteLine("Producer Task {0} cancelled", taskNum.ToString("000"))
ct.ThrowIfCancellationRequested()
End If
Thread.Sleep(sleepProducer)
Catch ex As OperationCanceledException
Console.WriteLine("Task " & taskNum.ToString("000") & " cancelling by request!")
Exit While
Catch ex As Exception
Console.WriteLine("Producer() Exception: " & ex.StackTrace)
End Try
If bc.Count >= itemsToProduce Then
moreItemsToAdd = False
End If
End While
sw.Stop()
' Let consumer know we are done.
Console.WriteLine("Producer stopped adding items! Added " & numAdded & " items in " & sw.Elapsed.TotalSeconds & " seconds!")
bc.CompleteAdding()
End Sub
#End Region
I know, I know... it looks complicated! But really it isn't. I'm not that smart! 1/2 the code is just to catch & handle the cancellation requests so that the processing doesn't corrupt any data. That, and a cheesy StopWatch() to time things... And yes, there are artifacts of earlier versions still commented out. Like I said "ROUGH"...
Line 17: Simply adds the itemId (our payload, could be anything) to the BlockingCollection (bc).
Line 20: If a cancellation, we take care of it here, and not a random part of the function, which would likely corrupt all kinds of things...
Line 31: I added this as a cheesy way to tell the Producers when to stop ...producing. This variable (limit) is set at the top of the code.
Line 38: bc.CompleteAdding() - This is a signal to everyone using the bc (BlockingCollection) that there will be no more items added. This way, the consumers know when to stop ...consuming!
"Why would they want to do that?"
Well, suppose you wanted a short-running task, or tasks, and needed to know they were done in order to continue... Yes, in my case, they are long-running, and in production I'll be starting each task with "TaskCreationOptions.LongRunning"
The Consumer() class is almost identical, with just a few tiny differences:
#Region "Consumer(s)"
Public Sub Consumer(ByVal taskNum As Integer, ByVal ct As CancellationToken)
If ct.IsCancellationRequested = True Then ' Was cancellation already requested?
Console.WriteLine("Consumer Task {0} was cancelled before Consumer thread created!", taskNum.ToString("000"))
ct.ThrowIfCancellationRequested()
End If
Dim totalTaken As Integer = 0
Dim sw As New Stopwatch
sw.Start()
While bc.IsCompleted = False
Dim itemOut As Integer = Nothing ' the payload
Try
itemOut = bc.Take()
Console.WriteLine("<-- " & taskNum.ToString("000") & " <-- [-1 => Q has: " & bc.Count & "] took: " & itemOut)
If ct.IsCancellationRequested Then
Console.WriteLine("Consumer Task {0} cancelled", taskNum.ToString("000"))
ct.ThrowIfCancellationRequested()
End If
totalTaken += 1
Catch ex As OperationCanceledException
Console.WriteLine("Task " & taskNum.ToString("000") & " cancelling by request!")
Exit While
Catch e As InvalidOperationException
' IOE means that Take() was called on a completed collection.
' In this example, we can simply catch the exception since the
' loop will break on the next iteration.
End Try
If (Not itemOut = Nothing) Then
Thread.Sleep(sleepConsumer)
End If
End While
sw.Stop()
If bc.IsCompleted = True Then
Console.WriteLine(vbCrLf & "Task " & taskNum.ToString("000") & " - No more items to take. Took " & totalTaken & " items in " & sw.Elapsed.TotalSeconds & " seconds!")
End If
End Sub
#End Region
End Module
Line 3: In both classes, we make sure to check right at the top to see if we've been cancelled. This way, we waste no time or resources if another Task/thread did the last piece of work just as we were being instantiated.
Line 13: itemOut = bc.Take() - Here we grab the next item (depends on FIFO or FILO/LIFO, as configured in the above discussion. This BlockingCollection does it all!
When you sit back and look at it, all the other code in this class is just to dress up Line 13!
So let's fire this puppy up!
Producer Task 001 (1) running!
--> 001 --> [+1 => Q has: 1] added: 1
<-- 100 <-- [-1 => Q has: 0] took: 1
Consumer Task 100 (2) running!
Consumer Task 101 (3) running!
Consumer Task 102 (4) running!
--> 001 --> [+1 => Q has: 1] added: 2
--> 001 --> [+1 => Q has: 2] added: 3
--> 001 --> [+1 => Q has: 3] added: 4
--> 001 --> [+1 => Q has: 4] added: 5
--> 001 --> [+1 => Q has: 5] added: 6
--> 001 --> [+1 => Q has: 6] added: 7
--> 001 --> [+1 => Q has: 7] added: 8
--> 001 --> [+1 => Q has: 8] added: 9
--> 001 --> [+1 => Q has: 9] added: 10
--> 001 --> [+1 => Q has: 10] added: 11
Producer stopped adding items! Added 11 items in 0.1631605 seconds!
<-- 101 <-- [-1 => Q has: 9] took: 2
<-- 100 <-- [-1 => Q has: 8] took: 3
<-- 101 <-- [-1 => Q has: 7] took: 4
<-- 102 <-- [-1 => Q has: 6] took: 5
creating producer 555...
Producer Task 555 (5) running!
<-- 100 <-- [-1 => Q has: 5] took: 6
Producer stopped adding items! Added 0 items in 1.09E-05 seconds!
<-- 101 <-- [-1 => Q has: 4] took: 7
<-- 102 <-- [-1 => Q has: 3] took: 8
cancelling random consumer...
<-- 100 <-- [-1 => Q has: 2] took: 9
<-- 101 <-- [-1 => Q has: 1] took: 10
<-- 102 <-- [-1 => Q has: 0] took: 11
Consumer Task 102 cancelled
Task 102 cancelling by request!
Task 102 - No more items to take. Took 2 items in 2.0128301 seconds!
Task 100 - No more items to take. Took 4 items in 4.0183264 seconds!
Task 101 - No more items to take. Took 4 items in 4.0007338 seconds!
cancelling random consumer...
creating consumer 222...
Task 222 - No more items to take. Took 0 items in 2.8E-06 seconds!
consumer Task 222 (6) running!
cancelling random producer...
cancelling random consumer...
Was this the output you expected?
Grab the entire solution 7z'd up for you, at the link below...
Download Solution from HERE!
It took me a while to figure out the whole CancellationToken concept, but now that I'm using it, and the bullet-proof-ness of the BlockingCollection, I'm confident my application can handle hundreds of objects per second without messing anything up.
My production application will read the amount of cores on the host machine, and use that to set the initial number of consumers. I will then adjust up & down, monitoring the time to complete (in an aggregate fashion), thus making the best of the host machine's resources, understanding that the host machine may be doing many other things at the same time as my application.
Thanks everyone!
I have an Excel sheet and its humongous.. Around like 200000 rows that needs to be processed.
All I have to do is read it and process them with a query on a DB2 table. I have written the program where its more than 8 hours to process 5000 rows.
Is there a way where I can simultaneously read the excel and execute the query. I want them to be independent of the process. I cannot use Parallel.for as reading and creating so many instance of threads is no advantage. ANy pipes and queues are of no use. THis is a dom method using and it does not read a row, it reads a string.. if there is a null value on the row, it executes the row and throws an null exception. I am well with Background workers and TPL's. Any idea or code would be appreciated. No DLL can be used apart from OPENXML
Ideally I do not want to add to array,, I want it in 2 diff variables and process them when read..
Read a row( only 2 columns, ignore other cols
create a thread to execute the row and in Parallel, execute the read row.
Merge into one single table.
display results.. Sounds simple but there are challenges.
.
Try
Using spreadsheetDocument As SpreadsheetDocument = spreadsheetDocument.Open(fileName, False)
Dim workbookPart As WorkbookPart = spreadsheetDocument.WorkbookPart
Dim worksheetPart As WorksheetPart = workbookPart.WorksheetParts.First()
Dim sheetData As SheetData = worksheetPart.Worksheet.Elements(Of SheetData)().First()
For Each r As Row In sheetData.Elements(Of Row)()
For Each c As Cell In r.Elements(Of Cell)()
Dim text As String
Try
text = c.CellValue.Text.ToString
Debug.Print(text.ToString)
If text IsNot Nothing AndAlso Trim(text).Length > 0 Then
Arr.Add(text.ToString)
End If
text = Nothing
j += 1
Catch
End Try
Next
text = Nothing
Next
End Using
Catch ex As Exception
MsgBox("Exception caught: " + ex.Message)
Debug.Print(ex.Message.ToString)
End
End Try
myArr = CType(Arr.ToArray(GetType(String)), String())
This is the process which is dividing the data into 2 parameters
For i As Integer = 2 To myArr.Count - 1 Step 2
If i And 1 Then
i = i - 1
Else
dstr = DateTime.FromOADate(Double.Parse(myArr(i).ToString())).ToShortDateString()
'Debug.Print(dstr.ToString & "----->" & i.ToString & "TCID--->" & myArr(i + 1).ToString)
DQueue.Enqueue(DateTime.FromOADate(Double.Parse(myArr(i).ToString())).ToShortDateString())
Tqueue.Enqueue((myArr(i + 1).ToString()))
TCArr.Add((myArr(i + 1).ToString()))
dc.Merge(ProcessQueryODBC(dstr, myArr(i + 1).ToString))
If dc.Rows.Count > 0 Then
dt.Merge(dc)
Else
nFound.Merge(CreateDT(dstr, myArr(i + 1).ToString()))
End If
End If
Next
Instead of opening a DB connection through ODBC. Can you export your data to a CSV file and then let DB2 perform the import?
somestring = "import from "myfile.csv" of DEL ...."
DoCmd.RunSQL somestring