I've read that using Try Catch blocks in SQL Server stored procedures can chow a lot of server resources. My question is just how much resources does it use?
I currently make use of Try Catch blocks and Transactions whenever my sproc is making a lot of data changes, which is great for preventing incorrect data from entering the db and also for logging the errors, but I'd like to make use of this method of programming in all my sprocs.
How much of a difference does it make?
Zero. Code using TRY/CATCH consumes exactly the same resources as code not using TRY/CATCH, only difference is that the former is usually more correct than the later. In fact TRY/CATCH code is more efficient in presence of errors as the code flow jumps straight to the catch block and avoids running the rest of the statements in the request/stored procedure only to roll back at the end.
Just for the record, I don't buy for a single second that writing code that checks for ##ERROR after each statement is even remotely a viable alternative.
I've read that in a book back when I was in university
For sure the reading did not refer to T-SQL code with TRY/CATCH, but instead was referring C++ code with or without exception (of JVM, or IL). Back in the dark ages there was a dispute about whether adding the exception handling code has performance impact (yes, it has) and whether we should consider this a factor (no, we shouldn't, code with exception handling has long since won that battle due to correctness). But this discussion is completely useless for your point of view: the back end engine that runs your T-SQL is compiled with exception handling and there is nothing you can do about it. Again, this has 0 impact on your T-SQL code.
Related
I’ve seen Objective C code using a collection of practices, raging from passing a pointer of NSError for execution finish status - to using NSAssert - to implementing #throw - to relaying on delegate for status code returned in the callback - to the old c method of returning a boolean/int indicating with 1 being success and co.
I can’t identify a consistent pattern for how should I be handling errors happening in my app running on client devices. For ex, what would you recommend handling for the following cases:
Client attempt to access a network resource, network resource timed out / returned 500?
Unexpected state that should have not even happened reached in logical code section?
Attempt to write to disk failed? (Out of disk space, not permission and code)
Coming from Java, server side practices exceptions the weapon of choice, using Objective C and C is seems that exceptions exist but are not encouraged. NSAssert seems harsh, as it will crash the application, which in most cases is not the optimal solution. So, I’d appreciate a Best practices advice.
Exceptions are used to indicate programmer error and/or non-recoverable errors only. Exceptions should not be used for flow control. NSAssert is more of a development tool. Use NSError for recoverable, user addressable (or caused) errors.
https://developer.apple.com/library/ios/documentation/Cocoa/Conceptual/ErrorHandlingCocoa/ErrorHandling/ErrorHandling.html
First we have to agree upon the terminology. I think there are two types of bad situations in a program: Errors and Exceptions.
Errors An unwanted situation that can happen but you can recover from it with the program memory being in a consistent state and further execution can continue.
eg1. Client attempt to access a network resource, network resource timed out / returned 500?
eg2. Attempt to write to disk failed? (Out of disk space, not permission and code)
Approach Handle the error the way you want it handled. Show user message, log it to file/console, report it to your back-end if you think it needs to be reported.
Exceptions A state that should never have been reached and may corrupt the memory state of the app in a way that prevents further execution.
This could happen at two levels.
Level 1 A logic error that you have overlooked and is now causing the exception.
eg1. accessing invalid array elements i.e. index out of bound exception.
In the above example, you unconsciously made a programming mistake. You couldn't have handled it because you overlooked it. This will crash the application.
Approach Your approach here should be to identify the bug and fix it. In the process some end users will be affected which is the case with the most meticulous software.
Level 2
As you are programming, you come across a situation that your business logic doesn’t permit.
eg1. Unexpected state that should have not even happened reached in logical code section?
eg2. A database entry that you always expect to be present as per your business logic. For example, let's say you are making an app that requires a currency table with currency names, unicode symbols for currencies and country names. The value from this table are to be shown as a drop down in your UI. You may create a table upon first launch of the app and insert the values or may be you ship the table in the bundle but you always expect it to have the values you are inserting. You make an sqlite select query->sqlite execution succeeds->but no values are returned. Business logic demands that values should be present but for some mysterious reason they are absent.
Here is where most confusion stems from because sometimes you can recover from some of these situations by treating them as errors and displaying a message to the end user. This approach is wrong. This is an exception at business logic level. One that you should prevent from happening.
Approach You should force crash the app here as well using NSAsserts both in development and production mode. This is an aggressive approach and some end users will be affected, but I have read experts advising to adopt an aggressive approach to finding bugs and remedying them rather than pretending they don’t exist or thinking you have smartly covered them all when all you have done is masking exceptions as errors through your “handling tactics”. I believe the experts are right. Coming from Java, you may find it a very objCeee.. way of doing things.
Nils and Bools You should understand that they are just facilitators to reach at the conclusion of whether you treat something as an error or exception. They are not the end in themselves. While writing your methods with return value, you should define what a nil or bool(no) signifies and document your intent on top of the method. Sometimes it would be an error, sometimes it would be an exception, sometimes it could be a normal expected result. While dealing with apple frameworks/third party libraries you should understand what their intent is when they return such values and determine how you want to treat them in your code.
Some other tips
Don’t use #try/catch because experts say so. If you wish to challenge them assuming that Apple has put it there for a reason, Apple is always right, experts(including Apple employees themselves) have got it wrong after all these years, you can go ahead and do it.
Formalize your approach to this->stick to it->document your intent. Don’t think it over and over. Because at some stage error/exceptions/nils/bools would cross the limits of propriety and reach the precincts of taste. You will be better off settling this once and for all and freeing up your time.
In a "religious" discussion about formatting of Microsoft T-SQL code, I questioned whether or not the GOTO statement was still available in T-SQL syntax. In 13 years of using T-SQL I have never had occasion to use it, and indeed didn't know if it existed. After a brief search of the documentation and to my consternation it does indeed exist!
My question is this:
Is there at least one case where GOTO statements would yield a solution that performs better than one in which other higher order programming constructs are used?
Encryption algorithm implementation
My question is NOT:
How do I use small bits of functionality without creating a whole function to do so?
How do I replicate error handling functionality without copy/paste?
I almost never use GOTO and can easily live without it.
The one case where I would consider using it is when I have complicated code that does lots of error checking. I might want to take some action when an error returns, and GOTO allows me to define a single block of code for the error checking.
You can solve this problem in multiple ways, but GOTO is a reasonable option that guarantees that errors are processed consistently and the code is not cluttered with a lot of if #Error = 0 . . . statements.
i saw goto a few times in large scripts where people used it to improve readability. sometimes it is better readable but mostly it turns into spaghetti code.
i see just one situation where goto can maybe perform better. its inside a multiple while loop. you can use goto once instead of multiple breaks which just exists the innermost loop. nevertheless in my opinion break is not much better than goto, its both not a good programming style.
while ...
while ...
while...
break
break
break
while ...
while ...
while...
goto endloops
endloops:
I've used GOTO statements mainly when SSIS was not available (don't ask!!) to provide some control flow to a large stored procedure used for ETL processing. That's the only time I've ever used them and while useful in that scenario, it's obviously not a scenario you'd be in.
First of all, thanks in advance for your help.
I've decided to ask for help in forums like this one because after several months of hard working, I couldn't find a solution for my problem.
This can be described as 'Why an object created in VB.net isn't released by the GC when it is disposed even when the GC was forced to be launched?"
Please consider the following piece of code. Obviously my project is much more complex, but I was able to isolate the problem:
Imports System.Data.Odbc
Imports System.Threading
Module Module1
Sub Main()
'Declarations-------------------------------------------------
Dim connex As OdbcConnection 'Connection to the DB
Dim db_Str As String 'ODBC connection String
'Sentences----------------------------------------------------
db_Str = "My ODBC connection String to my MySQL database"
While True
'Condition: Infinite loop.
connex = New OdbcConnection(db_Str)
connex.Open()
connex.Close()
'Release created objects
connex.Dispose()
'Force the GC to be launched
GC.Collect()
'Send the application to sleep half a second
System.Threading.Thread.Sleep(500)
End While
End Sub
End Module
This simulates a multithreaded application making connections to a MySQL database. As you can see, the connection is created as a new object, then released. Finally, the GC was forced to be launched. I've seen this algorithm in several forums but also in the MSDN online help, so as far as I am concerned, I am not doing anything wrong.
The problem begins when the application is launched. The object created is disposed within the code, but after a while, the availiable memory is exhausted and the application crashes.
Of course, this problem is hard to see in this little version, but on the real project, the application runs out of memory very quickly (due to the amount of connections made over the time) and as result, the uptime is only two days. Then I need to restart the application again.
I installed a memory profiler on my machine (Scitech .Net Memory profiler 4.5, downloadable trial version here). There is a section called 'Investigate memory leaks'. I was absolutely astonished when I saw this on the 'Real Time' tab. If I am correct, this graphic is telling me that none of the objects created on the code have been actually released:
The surprise was even bigger when I saw this other screen. According to this, all undisposed objects are System.Transactions type, which I assume are internally managed within the .Net libraries as I am not creating any object of this type on my code. Does it mean there is a bug on the VB.net Standard libraries???:
Please notice that in my code, I am not executing any query. If I do, the ODBCDataReader object won't be released either, even if I call the .Close() method (surprisingly enough, the number of unreleased objects of this type is exactly the same as the unreleased objects of type System.Transactions)
Another important thing is the statement GC.Collect(). This is used by the memory profiler to refresh the information to be displayed. If you remove it from the code, the profiler wont' update the real time diagram properly, giving you the false impression that everything is correct.
Finally, if you ommit the connex.Open() statement, the screenshot #1 will render a flat line (that means all the objects created have been successfully released), but unfortunatelly, we can't make any query against the database if the connection hasn't been opened.
Can someone find a logical explanation to this and also, a workaround for effectively releasing the objects?
Thank you all folks.
Nico
Dispose has nothing to do with garbage collection. Garbage collection is exclusively about managed resources (memory). Dispose has no bearing on memory at all, and is only relevant for unmanaged resources (database connections, file handles, gdi resource, sockets... anything not memory). The only relationship between the two has to do with how an object is finalized, because many objects are often implemented such that disposing them will suppress finalization and finalizing them will call .Dispose(). Explicitly Disposing() an object will never cause it to be collected1.
Explicitly calling the garbage collector is almost always a bad idea. .Net uses a generational garbage collector, and so the main effect of calling it yourself is that you'll hold onto memory longer, because by forcing the collection earlier you're likely to check the items before they are eligible for collection at all, which sends them into a higher-order generation that is collected less often. These items otherwise would have stayed in the lower generation and been eligible for collection when the GC next ran on it's own. You may need to use GC.Collect() now for the profiler, but you should try to remove it for your production code.
You mention your app runs for two days before crashing, and are not profiling (or showing results for) your actual production code, so I also think the profiler is in part misleading you here. You've pared down the code to something that produced a memory leak, but I'm not sure it's the memory leak you are seeing in production. This is partly because of the difference in time to reproduce the error, but it's also "instinct". I mention that because some of what I'm going to suggest might not make sense immediately in light of your profiler results. That out of the way, I don't know for sure what is going on with your lost memory, but I can make a few guesses.
The first guess is that your real code has try/catch block. An exception is thrown... perhaps not on every connection, but sometimes. When that happens, the catch block allows your program to keep running, but you skipped over the connex.Dispose() line, and therefore leave open connections hanging around. These connections will eventually create a denial of service situation for the database, which can manifest itself in a number of ways. The correction here is to make sure you always use a finally block for anything you .Dispose(). This is true whether or not you currently have a try/catch block, and it's important enough that I would say the code you've posted so far is fundamentally wrong: you need a try/finally. There is a shortcut for this, via a using block.
The next guess is that some of your real commands end up fairly large, possibly with large strings or image (byte[]) data involved. In this case, items end up on a special garbage collector generation called the Large Object Heap (LOH). The LOH is rarely collected, and almost never compacted. Think of compaction as analogous to what happens when you defrag a hard drive. If you have items going to the LOH, you can end up in a situation where the physical memory itself is freed (collected), but the address space within your process (you are normally limited to 2GB) is not freed (compacted). You have holes in your memory address space that will not be reclaimed. The physical RAM is available to your system for other processes, but over time this still results in the same kind of OutOfMemory exception you're seeing. Most of the time this doesn't matter: most .Net programs are short-lived user-facing apps, or ASP.Net apps where the entire thread can be torn down after a page is served. Since you're building something like a service that should run for days, you have to be more careful. The fix may involve significantly re-working some code, to avoid creating the large objects at all. That may mean re-using a single or small set of byte arrays over and over, or using streaming techniques instead of string concatenation or string builders for very large sql queries or sql query data. It may also mean you find this easier to do as a scheduled task that runs daily and shuts itself down at the end of the day, or a program that is invoked on demand.
A final guess is that something you are doing results in your connection objects still being in some way reachable by your program. Event handlers are a common source of mistakes of this sort, though I would find it strange to have event handlers on your connections, especially as this is not part of your example.
1 I suppose I could contrive a scenario that would make this happen. A simple way would be to build an object assumes a global collection for all objects of that type... the objects add themselves to the collection at construction and remove themselves at disposal. In this way, the object could not be collected before disposal, because before that point it would still be reachable... but that would be a very flawed program design.
Thank you all guys for your very helpful answers.
Joel, you're right. This code produces 'a leak' which is not necesarily the same as 'the leak' problem I have on my real project, though they reproduce the same symptoms, that is, the number of unreleased objects keep growing (and eventually will exhaust the memory) on the code mentioned above. So I wonder what's wrong with it as everything seems to be properly coded. I don't understand why they are not disposed/collected. But according to the profiler, they are still in memory and eventually will prevent to create new objects.
One of your guesses about my 'real' project hit the nail on the head. I've realized that my 'catch' blocks didn't call for object disposal, and this has been now fixed. Thanks for your valuable suggestion. However, I implemented the 'using' clause in the code in my example above and didn't actually fix the problem.
Hans, you are also right. After posting the question, I've changed the libraries on the code above to make connections to MySQL.
The old libraries (in the example):
System.Data.Odbc
The new libraries:
System.Data
Microsoft.Data.Odbc
Whith the new ones, the profiler rendered a flat line, whithout any further changes on the code, which it was what I've been looking after. So my conclussion is the same as yours, that is there may be some internal error in the old ones that makes that thing to happen, which makes them a real 'troublemaker'.
Now I remember that I originally used the new ones on my project (the System.Data and Microsoft.Data.Odbc) but I soon changed for the old ones (the System.Data.Odbc) because the new ones doesn't allow Multiple Active Recordsets (MARS) opened. My application makes a huge amount of queries against the MySQL database, but unfortunately, the number of connections are limited. So I initially implemented my real code in such a way that it made only a few connections, but they were shared accross the code (passing the connection between functions as parameter). This was great because (for example) I needed to retrieve a recordset (let's say clients), and make a lot of checks at the same time (example, the client has at least one invoice, the client has a duplicated email address, etc, which involves a lot of side queries). Whith the 'old' libraries, the same connection allowed to create multiple commands and execute different queries.
The 'new' libraries don't allow MARS. I can only create one command (that is, to execute a query) per session/connection. If I need to execute another one, I need to close the previous recordset (which isn't actually possible as I am iterating over it), and then to make the new query.
I had to find the balance between both problems. So I end up using the 'new libraries' because of the memory problems, and I recoded my application to not share the connections (so each procedure will create a new one when needed), as well as reducing the number of connections the application can do at the same time to not exhaust the connection pool.
The solution is far to ideal as it introduces spurious logic on the application (the ideal case scenario would be to migrate to SQL server), but it is giving me better results and the application is being more stable, at least in the early stages of the new version.
Thanks again for your suggestions, I hope you will find mines usefult too.
Cheers.
Nico
Adding Exceptional Handling is a good practise, but I have a doubt, In our database we have 10 SPs which are accessed by our data access layer, there is not much logic and the queries are also not big.The purpose of keeping the queries in SPs was to allow paged data to be easily passed to the presentation layer. So, my question is, should error handling be added to the SPs in this scenario when they do not include large complex queries?
yes.
TRY/CATCH will trap errors that will otherwise stop the stored proc running, for example, datatype conversion errors. It will also allow you to log the error in the CATCH block, for example, or retry (eg have proc call itself)
Most paging queries should be a single SELECT statement, or maybe two if you're optimizing with SET ROWCOUNT/LIMIT. I don't think that there's any error that could possibly occur in this instance that you would be able to handle from within the database.
Generally the main reason for implementing error handling at the database level is to combine it with transactional logic so that you roll back the right changes at the right time when necessary. If you aren't actually executing any DML, I don't really see a good reason to write special error-handling code.
Of course you haven't shown us the queries here either... if you are writing some kind of multi-statement DML in there, then putting aside the fact that it's a weird thing to be doing in a paging query, you would want to have some sort of error-handling. It's not so much the size of the query that matters, but rather what the query does; if you're making modifications to the data then you should consider implementing error handling, but if not, then I don't see any good reason for it.
I have worked with SQL for several years now, primarily MySQL/PhpMyAdmin, but also Oracle/iSqlPlus and PL/SQL lately. I have programmed in PHP, Java, ActionScript and more. I realise SQL isn't an imperative programming language like the others - but why do the error messages seem so much less specific in SQL? In other environments I'm pointed straight to the root of the problem. More often that not, MySQL gives me errors like "error AROUND where u.id = ..." and prints the whole query. This is even more difficult with stored procedures, where debugging can be a complete nightmare.
Am I missing a magic tool/language/plugin/setting that gives better error reporting or are we stuck with this? I want a debugger or language which gives me the same amount of control that Eclipse gives me when setting breakpoints and stepping trough the code. Is this possible?
I think the answer lies in the fact that SQL is a set-based language with a few procedural things attached. Since the designers were thinking in set-based terms, they didn't think that the ordinary type of debugging that other languages have is important. However, I think some of this is changing. You can set breakpoints in SQL Server 2008. I haven't used it really as you must have SQL Server 2008 databases before it will work and most of ours are still SQL Server 2000. But it is available and it does allow you to step through things. You still are going to have problems when your select statement is 150 lines long and it knows that the syntax isn't right but it can't point out exactly where as it is all one command.
Personally when I am writing a long procedural SP, I build in a test mode that includes showing me the results of things I do, the values of key variables at specific points I'm interested in, and print staments that let me know what steps have been completed and then rolling the whole thing back when done. That way I can see what would have happened if it had run for real, but not have hurt any of the data in the database if I got it wrong. I find this very useful. It can vastly increase the size of your proc though. I have a template I use that has most of the structure I need set up in it, so it doesn't really take me too long to do. Especially since I never add an insert. update or delete to a proc without first testing the associated select to ensure I have the records I want.
I think the explanation is that "regular" languages have much smaller individual statements than SQL, so that single-statement granularity points to a much smaller part of the code in them than in SQL. A single SQL statement can be a page or more in length; in other languages it's usually a single line.
I don't think that makes it impossible for debuggers / IDEs to more precisely identify errors, but I suspect it makes it harder.
I agree with your complaint.
Building a good logging framework and overusing it in your sprocs is what works best for me.
Before and after every transaction or important piece of logic, I write out the sproc name, step timestamp and a rowcount (if relevant) to my log table. I find that when I have done this, I can usually narrow down the problem spot within a few minutes.
Add a debug parameter to the sproc (default to "N") and pass it through to any other sprocs that it calls so that you can easily turn logging on or off.
As for breakpoints and stepping through code, you can do this with MS SQL Server (in my opinion, it's easier on 2005+ than with 2000).
For the simple cases, early development debugging, the sometimes cryptic messages are usually good enough to get the error resolved -- syntax error, can't do X with Y. If I'm in a tough sproc, I'll revert to "printf debugging" on the sproc text because it's quick and easy. After a while with your database of choice, the simple issues become old hat and you just take them in stride.
However, once the code is released, the complexity of the issues is way too high. I consider myself lucky if I can reproduce them. Also, the places where the developer in me would want a debugger the DBA in me says "no way you're putting a debugger there."
I do use the following tactics.
During writing of the stored procedure have a #procStep var
each time a new logical step is executed
set #procStep = "What the ... is happening here " ;
the rest is here