I have a loop of several hundred items which need to be processed.
Each item is processed by conditionally setting a global SQLConnection where upon the item is processed using this SQLConnection as part of the processing.
For this reason it is vital that none of these items is allowed to be processed in parallel.
I appreciate that this is not good design and I hope to rectify it as soon as is practical.
However it would seem that despite my best efforts, this code is experiencing some form of multi-threading. Somehow one of these tasks has thrown an exception.
This exception is the violation of a foreign key constraint, but indicates that it was operating against a SQLConnection which it has no business connecting to.
Naturally I have concerns about this, however to my knowledge there is no multi threading code in this app.
I wonder Is it possible to introduce multi threading without explicitly creating new threads
EDIT:
VB.Net 3.5SP1
Console App + Class Libraries
Occasionally Calls out to web services
Makes SQL calls
not much of anything else. No Winforms, no WPF.
Yes - using System.Timers.Timer and/or System.Threading.Timer can cause the effect your describing. Whenever a timer ticks a new work item is queued in the ThreadPool - so essentially you have a multi threading program without explicitly creating new threads.
If the timer is AutoReset (remains enabled after elapsed has been called) you might cause another call to the same handler concurrently.
In addition to the others that have been mentioned: parallel extensions (PLINQ and task parallel library).
Alternatively tasks (ie Task objects) are not called threads, but are. Tasks are commonly found near lambda expressions, check if you have any.
Oh, and async sockets too and all the other async IOs.
BUT:
Instead of trying to avoid multithreading at all cost, wouldn't it be easier to lock ? Sorry if the question is naive, I may miss something.
Could it be that your code is called from a 3rd party library. By using events another library can call your code - from as many threads as it like.
I suggest you check the code that invoke the code that changes and make sure that there's no suspicious calls to your code.
Related
I have a program I'm writing in vb.net that has ballooned into the most complicated thing I've ever written. Because of some complex math and image rendering that's happening constantly I've been delving into multithreading for the first time to improve overall performance. Things have honestly been running really smoothly, but we've just added more functionality that's causing me some trouble.
The new functionality comes from a pair of DLLs that are each processing a video stream from a USB camera and looking for moving objects. When I start my program I initiate the DLLs and they start viewing the cameras and processing the videos. I then periodically ping them to see if they have detected anything. This is how I start and stop them:
Declare Function StartLeftCameraDetection Lib "DetectorLibLeft.dll" Alias "StartCameraDetection" () As Integer
Declare Function StopLeftCameraDetection Lib "DetectorLibLeft.dll" Alias "StopCameraDetection" () As Integer
When I need to check if they've found any objects I use several functions like this:
Declare Function LeftDetectedObjectLeft Lib "DetectorLibLeft.dll" Alias "DetectedObjectLeft" () As Integer
All of that works really well. The problem is, I've started to notice some significant lag in my UI and I'm thinking it may be coming from the DLLs. Forgive my ignorance on this, but as I said I'm new to using multiple threads (and incorporating DLLs too if I'm honest). It seems to me that when I start a DLL it running it's background tasks on my main thread and just waiting for me to ping it for information. Is that the case? If so, is it possible to have the DLL running on a sperate thread so it doesn't affect my UI?
I've tried a few different things but I can't seem to address the lag. I moved the code that pings the DLL and processes whatever information it gets into a sperate thread, but that hasn't made any difference. I also tried calling StartLeftCameraDetection from a separate thread but that didn't seem to help either. Again, I'm guessing that's because the real culprit is the DLL itself running these constant background tasks on my main thread no what thread I actually call it's functions from.
Thanks in advance for any help you might be able to offer!
There's a lot to grok when it comes to threading, but I'll try to write a concise summary that hits the high points with enough details to cover what you need to know.
Multi-threaded synchronization is hard, so you should try to avoid it as much as possible. That doesn't mean avoiding multi-threading at all, it just means avoiding doing much more than sending a self-contained task off to a thread to run to completion and getting the results back when it's done.
Recognizing that multi-threaded synchronization is hard, it's even worse when it involves UI elements. So in .NET, the design is that any access to UI elements will only occur through one thread, typically referred to as the UI thread. If you are not explicitly writing multi-threaded code, then all of your code runs on the UI thread. And, while your code is running, the UI is blocked.
This also extends to external routines that you run through Declare Function. It's not really accurate to say that they are doing anything with "background tasks on the main thread", if they are doing anything with "background tasks" they are almost certainly implementing their own threading. More likely, they aren't doing any task breakdown at all, and all of their work is being done on whichever thread you use to call them---the UI thread if you're not doing anything else.
If the work being done in these routines is CPU-bound, then it would definitely make sense to push it off onto a worker thread. Based on your comments on what you already tried:
I moved the code that pings the DLL and processes whatever information it gets into a sperate thread, but that hasn't made any difference. I also tried calling StartLeftCameraDetection from a separate thread but that didn't seem to help either.
I think the most likely problem is that you're blocking in the UI thread waiting for a result from the background thread.
The best way to avoid this depends on exactly what the routines are doing and how they produce results. If they do some sort of extended process and return everything in function results, then I would suggest that using Await would work well. This will basically return control to the UI until the operation finishes, then resume whatever the rest of the calling routine was going to do.
Note that if you do this, the user will have full interaction with the UI, and you should react accordingly. You might need to disable some (or all) operations until it's done.
There are a lot of resources on Async and Await. I'd particularly recommend reading Stephen Cleary's blog articles to get a better understanding of how they work and potential pitfalls that you might encounter.
New to NServiceBus (4.7.5) and just implemented an NSB host.exe hosted service (implementing IWantToRunWhenBusStartsAndStops) that detects changes to database tables and notifies subscribing web apps by publishing events, e.g. "CustomerDataWasUpdatedEvent". In the future we will perform the actual update through messagehandlers receiving commands obviously, but at the moment this publishing service just polls the database etc.
It all works well, however, approaching production, I noticed that David Boike, in his latest edition of "Learning NServiceBus", states that classes implementing
IWantToRunWhenBusStartsAndStops are really mostly for development and rarely used in production. I set up my database change detection in the Start method and it works nicely, does anyone know why this is discouraged?
Here is the comment in the actual book:
https://books.google.se/books?id=rvpzBgAAQBAJ&pg=PA110&lpg=PA110&dq=nservicebus+iwanttorunwhenbusstartsandstops+in+production+david+boike&source=bl&ots=U6sNII0nm3&sig=qIXffOVFhcy-_3qDnSExRpwRlD4&hl=sv&sa=X&ei=lHWRVc2_BKrWywPB65fIBw&ved=0CBsQ6AEwAA#v=onepage&q=nservicebus%20iwanttorunwhenbusstartsandstops%20in%20production%20david%20boike&f=false
The actual quote is:
...it isn't common to have widespread use of in a production system.
Uncommon is not the same thing as discouraged.
That said I do think there is intent here by the author to highlight the fact that further up the page they assert that this is not a good place to be doing lots of coding, as an unhandled exception can cause the whole process to fail.
The author actually does go on to mention a possible use case for when you may want to load a resource(s) to do work within the handler.
Ok, maybe it's just this scenario we have that is a bit uncommon
Agreed - there is nothing fundamentally wrong with your approach. I recently did the same thing as you for wiring up SqlDependency to listen for database events and then publish a message as a result. In these scenarios there is literally nothing else you can do other than to use IWantToRunAtStatup.
Also, David himself often trawls the nservicebus tag, maybe he'll provide a more definitive answer than mine.
I'll copy the answer I gave in the Particular Software Google Group...
I'll quote myself directly here:
An implementation of IWantToRunWhenBusStartsAndStops is a great place to create a quick interface in order to test messages during debugging by allowing you to send messages based on the console input. Apart from this, it isn't common to have widespread use of them in a production system. One possible production use case will be to provision a resource needed by the endpoint at startup and then tear it down when the endpoint stops.
I think if I could add a little bit of emphasis it would be to "widespread use". I'm not trying to say you won't/can't have an IWantToRunWhenBusStartsAndStops in production code or that avoiding them is a best practice. I am trying to say that having a ton of them is probably a code smell.
Above that paragraph in the book, I warn about IWantToRunWhenBusStartsAndStops not having any ambient transactions or try/catch stuff going on. THAT is really the key part. If you end up throwing an exception in an IWantToRunWhenBusStartsAndStops, tyou can run into big problems. If you use something like a .NET Timer and then throw an exception, you can crash your process!
Let me tell you how I screwed up on this in my first-ever NServiceBus system. The system (still in use today, from what I hear) is responsible for ingesting more than 3000 RSS feeds (probably a lot more than that now) into a CMS. So processing each feed, breaking it up into items, resizing images, encoding attached video for mobile ... all those things were handled in NServiceBus message handlers, which was scaled out to multiple servers, and that was all fantastic.
The problem was the scheduler. I implemented that as an IWantToRunWhenBusStartsAndStops (well, actually IWantToRunAtStartup at that time) and it quickly turned into a mess. I kept the whole table worth of feed information in memory so that I could calculate when to fire off the next ProcessFeed command. I was using the .NET Timer class, and IIRC, I eventually had to use threading primitives like ManualResetEvent in order to coordinate the activity. And because I was using .NET Timer, if the scheduler threw an exception, that endpoint failed and had to restart. Lots of weird edge cases and it was always a quagmire of bugs. Plus, this was now a singleton "commander app" so while the feed/item processors could be scaled out, the scheduler could not.
As I got more experienced with NServiceBus, I realized that each feed should have been a saga, starting from a FeedCreated event, controlled through PauseProcessing and ResumeProcessing commands, using timeouts to control the next processing time, and finally (perhaps) ended via a FeedRemoved event. This would have been MUCH more straightforward and everything would have executed inside transactionally-controlled message handlers.
That experience led me to be a little bit distrustful/skeptical of IWantToRunWhenBusStartsAndStops. Not saying it's bad, just something to be aware of. Always be prepared to consider if what you're trying to do couldn't be better accomplished in another way.
I'm learning VB.NET coming from a VB6 and Java background.
In my app, I've got a function that validates the fields on a form. All it is doing is reading them, not updating. I've searched and see info about the backgroundWorker class, but all the examples are about updating the fields.
I understand the idea of threading and how it works, but have never written code that spawned threads myself. I've always let the language handle it. It seems like a lot of work that I would have to write a sub using the backgroundWorker for every time I wanted to read or update each field. The couple of books I've got that introduce you to the language show you reading or updating the field directly.
How do I know what threads are running other than writing the code like I'm used to then running through debugger to figure out what variables are on which thread?
Thanks.
Here and here is some reading on the BackgroundWorkerProcess. My advice, don't use this unless you have to i.e. only when you have a long running process and want to
Have the user switch between screens while that task is running.
Use a progress indicator on the form
That being said, I find it useful in cases like processing invoices. When I have to generate say, 4k invoices, while that task is running I can put an indacator on the form.
I find the following book helpful "Visual Basic 2008 Recipes" in explaining several use of threading, including the BackGroundWorker
The background worker does a lot of work for you. Certainly easier than managing threads and marshalling callbacks yourself. However, I agree with Saif... no point in doing any work unless there is some benefit to be had. Use it only for processes that may potentially take a lot of time.
Hopefully you're using VS2010, as it added some threading features. For example, use the Debug Location toolbar to select the thread of interest.
I am having an issue where I need to implement a type of thread scheduling... I'm looking to implement a list of tasks, each with a set time that they need to execute, and after the time is up they will execute the respective code that goes with that task. I would use a timer based solution, however, I don't think that it would be very efficient. There will be a very large list of tasks, some of which need to be executed within seconds of being placed in the list.
To an extent, I am familiar with multithreading and expect that to be used here; I'm just looking for a starting place - someone suggested a thread pool but I'm not too sure if you can assign timers to those.
Any help/suggestions are greatly appreciated!
I ended up creating a class that inherits System.Timers.Timer. I read that there are three types of timer classes and the one I chose is enhanced for multithreaded solutions.
I have a legacy WinForms Mdi App in VB.Net 2.0 which I am adding functionality to. One of the additions is a warning which needs to be raised when the current time nears a specified value (a deadline). My intention is to check the time once an hour until there is less than an hour until the deadline, then display warnings at specified intervals until the time's up.
The user needs to be able to continue to use the app up to and even after the deadline, but they need to periodically be made aware of the deadline's proximity.
The app does not use System.Threading yet and my knowledge of it is limited at this time. I do know that there are 3 different Timer() methods available:
System.Threading.Timer(),
Windows.Forms.Timer() and
System.Timers.Timer()
My question is, which is the best way to go with this? I attempted to use the threaded timer, but since WinForms are not thread safe I got a run time error trying to access another class. Is it worth making the class/form thread safe? Am I completely off track?
Thanks.
This article explains pretty well:
Comparing the Timer Classes in the .NET Framework Class Library
It sounds like System.Windows.Forms.Timer is the one for you.
My guideline: If you want the timer to run on your main GUI thread, stick with Windows.Forms.Timer. If it's okay for your timer to be called asynchronously on a thread pool thread, or if you don't want to experience the small delays that System.Windows.Forms.Timer tends to suffer, use System.Timers.Timer. System.Threading.Timer has a different interface from the other two and is not thread-safe; personally, I'm not a fan.
I would just use the Forms timer. I think I read that it's not as accurate, but it sounds like you don't need it to be.
I agree that Windows.Forms.Timer() is the best for this case as it handles the cross thread marshalling issues.
Some useful related links:
Windows Presentation Foundation Threading Model
WinForms UI Thread Invokes: An In-Depth Review of Invoke/BeginInvoke/InvokeRequred
The System.Forms.Timer actually works on the main thread using the windows message queue. This makes it somewhat inacurate but since you don't really need ms precision it's good enough.
You could use one of the other timers that work on a separate thread but since you need to activate a winforms component that work in the main you'll need to use Form.Invoke or some other way to pass the event to the main thread - which would cause some latency as well.
In conclusion use the System.Forms.Timer when you need to activate a winforms based component.
Ok, first things first...
If you want to show the user a form and do something in the background I would use the BackgroundWorker class, it worked for me before.
Also, you need invoke methods as mentioned before and as Chris said, it sounds harder than what it actually is.
Here's a link which I think will help you out.
http://msdn.microsoft.com/en-us/library/ms171728(VS.80).aspx