Should I avoid fuzzing the error handling code - testing

Error handling code plays an important role in programming, hence, when I test the program with afl, should I pay attention to the error handling code part, or should I avoid this part from fuzzing.

Error handling code can certainly contain bugs, so there's no reason to keep it out. You may want to configure your instrumentation to ignore common error results or error conditions.

Related

How to find specific implicit conversion in VS2013? [duplicate]

When building an application in Visual Studio. It shows the following error when exceeding the error count
fatal error C1003: error count exceeds 100; stopping compilation
Is there a way to increase the error limit?
This limitation is hardcoded. Here is the post from the MSFT employee in the microsoft.public.vsnet.general group dated 2006 (look for 'Fatal Error C1003'):
Hi,
Unfortunately this 100 limitation is
hard coded and cannot be changed. It's
just inpractical to keep all errors
information around since one error may
cause other several errors.
I hope you understand the rational
behind this design by our product
team. However, if you still have
concerns about this, please feel free
to submit your feedback at
http://connect.microsoft.com/Main/content/content.aspx?ContentID=2220
which is monitored by our product
team. Thank you for your
understanding.
Sincerely, Walter Wang
(waw...#online.microsoft.com, remove
'online.') Microsoft Online Community
Support"
I don't think so. VS basically reports all errors it encounters during compilations. There might be some erroneous parts of the code that make the compiler getting caught in an infinite "error" loop.
The limit was implemented to avoid that. In most cases the 100 errors you get are just the same error reported over and over again. What would be the sense in increasing the number of repetitions?
Maybe you can post the code snippet where the error occurs first, so we can help you fix it.
I believe that it is a hard-coded limit, so no.
As others have commented, it's difficult to understand what you want to achieve by this.
At the end of the day, you'll have to fix them all, so get stuck in and start fixing them. Eventually, you'll get below 100, and you can start counting them.
It is not normally valuable to report the actual number of errors when this occurs. Most of the time, when you get C1003, it's actually only a few real errors, leading to a massive chain of other errors.
(e.g.)
If there is an error in a .h file, that error will be reported in every .cpp file that #includes it.
If there is an error that prevents any kind of identifier being defined (e.g. a class, variable, method name), then every time you try to use it later on, an error will be reported.
Workaround to reduce number of reported errors:
rename cl.exe to cl-orig.exe
roll your own cl.exe that launches cl-orig.exe, capturing its stdout / stderr
parse stderr, looking for error messages and counting them
breaks after first n errors
See http://msdn.microsoft.com/en-us/library/ms682499(v=vs.85).aspx for some hints.
I also have a project like this: sometimes Visual Studio decides there is a lot to do, emits 100 really irrelevant messages about other parts of the solution and aborts the build because it reached the message limit without working on the project I'm interested on.
The workaround we have found is to use msbuild to build the solution from a command prompt: the Use MSBuild walkthrough outlines the steps. msbuild outputs all messages to the console and once the build completes we can work and debug again in Visual Studio. Not ideal, but it lets us complete the task at hand.

Should app crash or continue on normally while noting the problem?

Options:
1) When there is bad input, the app crashes and prints a message to the console saying what happened
2) When there is bad input, the app throws away the input and continues on as if nothing happened (though nothing the problem in a separate log file).
While 2 may seem like the obvious solution, the app is an engine and framework for game development, so if a user is writing something and does something wrong, it may be beneficial for that problem to be immediately obvious (app crashing) rather than it being ignored and the user potentially forgetting to check the log to see if there were any problems (may forget if the programmed behavior isn't very noticeable on screen, so he doesn't catch that it is missing).
There is no one-size-fits-all solution. It really depends on the situation and how bad the input is.
However, since you specifically mentioned this is for an engine or framework, then I would say it should never crash. It should raise exceptions or provide notable return codes or whatever is relevant for your environment, and then the application developer using your framework can decide how to handle. The framework itself should not make this decision for all apps that utilize the framework.
I would use exceptions if the language you are using allows them..
Since your framework will be used by other developers you shouldn't really constraint any approach, you should let the developers catch your exception (or errors) and manage what to do..
Generally speaking nothing should crash on user input. Whether the app can continue with the error logged or stop right there is something that is useful to be able to configure.
If it's too easy to ignore errors, people will just do so, instead of fixing them. On the other hand, sometimes an error is not something you can fix, or it's totally unrelated to what you're working on, and it's holding up your current task. So it depends a bit on who the user is.
Logging libraries often let you switch logs on and off by module and severity. It might be that you want something similar, to let users configure the "stop on error" behaviour for certain modules or only when above a certain level of severity.
Personally I would avoid the crash approach and opt for (2) that said make sure that the error is detected and logged and above all avoid any swallowing of errors (e.g. empty catch).
It is always helpful to have some kind of tracing/logging module, for instance later when you are doing performance tuning or general troubleshooting.
It depends on what the problem is. When I'm programming and writing error handling I use this as my mantra:
Is this exception really exceptional?
Meaning, is the error in input or whatever condition is "not normal" recoverable? In the case of a game, a File not Found exception on a texture could be recoverable and you could show a default texture so you know something broke.
However, if you have textures in a compressed file and you keep getting checksum errors, that would be an exceptional exception and I would crash the game with the details.
It really boils down to: can the application keep running without issue?
The one exception to this rule though (ha ha) is, if something is corrupted you can no longer trust your validation methods and you should crash as quickly as you can to prevent the corruption from spreading.

How to refactor VB6 code to prevent run-time errors

A VB6 app is experiencing a run-time errors at a variety of places.
I know this is the result of poor error handling, but is it possible to analyse the code to see where it is susceptible to run-time errors?
Any application is susceptible to run-time errors where there is no error handling around calls to external resources, so you could identify those points as a start.
I've used a free-tool (many years ago) that could retro-fit error handling to VB6 code, which would at least log errors and the point that they occurred.
Here it is: The HuntErr Addin for easy error handling in VB6
You need to make sure that every one of the methods (functions, subs, properties...) in your code base has an error handling statement. It's probably true that not every single one can generate a run time error, but that will protect the application from crashing without a lot of upfront analysis.
Make sure there's a statement before any executable line of code that says "On Error GoTo..." with a label, and then make sure to put that label with some error handling code at the bottom of the method. I've used a free tool called MZ-Tools 3.0 that allows you to automate the inclusion of this text. There is an Error Handler tab in the options that lets use specify what text you want to put in and where. This is what mine looks like:
On Error GoTo l{PROCEDURE_NAME}_Error
{PROCEDURE_BODY}
Exit {PROCEDURE_TYPE}
l{PROCEDURE_NAME}_Error:
LogError "{MODULE_NAME}", "{PROCEDURE_NAME}", Err, Err.Description
Then I just make sure that the LogError function exists and writes the error out to a log file that I can review.
Common sources of run-time errors in VB6 apps include
Accessing a key in a collection that doesn't exist
Calling a method or property on an object that is nothing
Using CLng to convert a string to a number when the string is null
Accessing an array beyond its length (like after calling Split and then assuming that the string has the number of pieces you expected)
So besides doing what others have suggested and analyzing where the actual errors are coming from, you could start by looking for areas such as these in your code and putting appropriate error handling around them. Keep in mind that often the best "error handling" doesn't involve using On Error at all, but preventing the error ahead of time by checking for these boundary case, like
If Not Object Is Nothing
If Len(string) > 0
If UBound(array) > x
etc...
There are some good answers here with both the On Error GoTo recommendations and the common errors that fall through the cracks that bwarner mentions.
But maybe widen the scope and utilize built-in tools to analyze code like breakpoints, watch expressions, and especially good for debugging run-time errors, the locals window (often overlooked in debugging, but very powerful) and the call stack. You can get a lot of great info on that from here: Debugging Your Code and Handling Errors
Other things to think about that may be helpful:
Invest in a tool that will help you
with the analysis like CodeSMART
2009 for VB6 or VB Project
Analyzer.
Try to port the existing application
to VB.NET - not to actually port and
use, but to view the conversion log
for things that need to be fixed.
Following Ryan's answer and the comment in response, you don't have to put error handling in every routine, just every Event and Sub Main() (and API callbacks if they don't already have it).
API callbacks refer to routines called directly by the Win32API, most often passed to Declared functions using AddressOf. (I.e. search your code for AddressOf and ensure all routines mentioned as arguments have error handlers that catch errors and do not allow them to attempt to bubble up.)
And I've just noticed this doesn't really answer the original question asked (although given the comment in response to Ryan's answer it is a good first step): Once you have error handling in every Event, etc, you will catch all errors, but you won't be able to directly analyse where all errors occur. You will need to extend the error logging to at least all the routines called by the events that log errors to more accurately locate the exact source of each error.
In VB6 a Runtime error occurs exactly when an Event function is called without error handling. So at least all your event handling functions (like Form.Open()) should be surrounded by an error handler (yes, I know VB6 does not have them), which can nicely be implemented like this (We do it in all our applications like this):
Use this as the first line of EVERY event handling function (it is a large valid label which sets the On Error at the end):
¦¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯: On Error GoTo ErrorHandler:
Now use this on the end of all those functions:
¦____________________________________________________________________________________________________________________________________: Exit Sub
ErrorHandler: handleError CodeDb, "Form_frm_filter_deletecolum", "cmd_deletecolum_Click", err.Number, err.description, err.Source, err.LastDllError, err.Helpfile, err.HelpContext, Erl: err.Clear
But replace the two strings with the module name and function name. AND replace each On Error Goto 0 with On Error Goto ErrorHandler.
Now create a function handleError with the given arguments (in my app it automatically sends a bug report into our bugtracking system), and display a nice error message.
We even pushed this a bit further, by having a prebuilt process that adds similar lines to all other (means non-event functions, or functions just called by other functions), to remember the line in which the error occured and to accumulate a complete stack trace (Yeah, stacktraces in VB6!). In addition this process adds line numbers to each line so they are given in the Erl part of the error handler.
Our tool is written as some MS Access modules, so I can't simply provide it for you, but you see where you have to go for it.

Should examples--even beginner examples--include the error-handling code?

Brian Kernighan was asked this question in a recent interview. I'll quote his reply:
Brian: I'm torn on this. Error-handling code tends to be bulky and very uninteresting and uninstructive, so it often gets in the way of learning and understanding the basic language constructs. At the same time, it's important to remind programmers that errors do happen and that their code has to be able to cope with errors.
My personal preference is to pretty much ignore error handling in the earlier parts of a tutorial, other than to mention that errors can happen, and similarly to ignore errors in most examples in reference manuals unless the point of some section is errors. But this can reinforce the unconscious belief that it's safe to ignore errors, which is always a bad idea.
I often leave off error handling in code examples here and on my own blog, and I've noticed that this is the general trend on Stack Overflow. Are we reinforcing bad habits? Should we spend more time polishing examples with error handling, or does it just get in the way of illustrating the point?
I think it might be an improvement if when posting example code we at least put comments in that say you should put error handling code in at certain spots. This might at least help somebody using that code to remember that they need to have error handling. This will keep the extra code for error handling out but will still reinforce the idea that there needs to be error handling code.
Any provided example code will be copy-pasted into production code at least once, so be at your best when writing it.
Beyond the question of cluttering the code when you're demonstrating a coding point, I think the question becomes, how do you choose to handle the error in your example code?
That is to say, what do you do ? What's fatal for one application is non-fatal for another. e.g. if I can't retrieve some info from a webserver (be it a 404 error or a non-responsive server) that may be fatal if you can't do anything without that data. But if that data is supplementary to what you're doing, then perhaps you can live without it.
So the above may point to simply logging the error. That's better than ignoring the error completely. But I think often the difficulty is in knowing how/when (and when not) to recover from an error. Perhaps that's a whole new tutorial in itself.
Examples should be illustrative. They should always show the point being made clearly with as little distraction as possible. Here's a meta-example:
Say we want to read a number from a file, add 3, and print it to the console. We'll need to demonstrate a few things.
infile = file("example.txt")
content = infile.read()
infile.close()
num = int(content)
print (3 + num)
wordy, but correct, except there are a few things that could go wrong. First, what if the file didn't exist? What if it does exist but doesn't contain a number?
So we show how the errors would be handled.
try:
infile = file("example.txt")
content = infile.read()
infile.close()
num = int(content)
print (3 + num)
except ValueError:
print "Oops, the file didn't have a number."
except IOError:
print "Oops, couldn't open the file for some reason."
After a few iterations of showing how to handle the errors raised by, in this case, file handling and parsing. Of course we'd like to show a more pythonic way of expressing the try clause. Now we drop the error handling, cause that's not what we're demonstrating.
First lets eliminate the unneeded extra variables.
infile = file("example.txt")
print (3 + int(infile.read()))
infile.close()
Since we're not writing to it, nor is it an expensive resource on a long-running process, it's actually safe to leave it open. It will closewhen the program terminates.
print ( 3 + int(file("example.txt").read()))
However, some might argue that's a bad habit and there's a nicer way to handle that issue. We can use a context to make it a little clearer. of course we would explain that a file will close automatically at the end of a with block.
with file("example.txt") as infile:
print (3 + int(infile.read()))
And then, now that we've expressed everything we wanted to, we show a complete example at the very end of the section. Also, we'll add some documentation.
# Open a file "example.txt", read a number out of it, add 3 to it and print
# it to the console.
try:
with file("example.txt") as infile:
print (3 + int(infile.read()))
except ValueError: # in case int() can't understand what's in the file
print "Oops, the file didn't have a number."
except IOError: # in case the file didn't exist.
print "Oops, couldn't open the file for some reason."
This is actually the way I usually see guides expressed, and it works very well. I usually get frustrated when any part is missing.
I think the solution is somewhere in the middle. If you are defining a function to find element 'x' in list 'y', you do something like this:
function a(x,y)
{
assert(isvalid(x))
assert(isvalid(y))
logic()
}
There's no need to be explicit about what makes an input valid, just that the reader should know that the logic assumes valid inputs.
Not often I disagree with BWK, but I think beginner examples especially should show error handling code, as this is something that beginners have great difficulty with. More experienced programmers can take the error handling as read.
One idea I had would be to include a line like the following in your example code somewhere:
DONT_FORGET_TO_ADD_ERROR_CHECKING(); // You have been warned!
All this does is prevent the code compiling "off the bat" for anyone who just blindly copies and pastes it (since obviously DONT_FORGET_TO_ADD_ERROR_CHECKING() is not defined anywhere). But it's also a hassle, and might be deemed rude.
I would say that it depends on the context. In a blog entry or text book, I would focus on the code to perform or demonstrate the desired functionality. I would probably give the obligatory nod to error handling, perhaps, even put in a check but stub the code with an ellipsis. In teaching, you can introduce a lot of confusion by including too much code that doesn't focus directly on the subject at hand. In SO, in particular, shorter (but complete) answers seem to be preferred so handling errors with "a wave of the hand" may be more appropriate in this context as well.
That said, if I made a code sample available for download, I would generally make it as complete as possible and include reasonable error handling. The idea here is that for learning the person can always go back to the tutorial/blog and use that to help understand the code as actually implemented.
In my personal experience, this is one of the issues that I have with how TDD is typically presented -- usually you only see the tests developed to check that the code succeeds in the main path of execution. I would like to see more TDD tutorials include developing tests for alternate (error) paths. This aspect of testing, I think, is the hardest to get a handle on since it requires you to think, not of what should happen, but of all the things that could go wrong.
Error handling is a paradigm by itself; it normally shouldn't be included in examples since it seriously corrupts the point that the author tries to come across with.
If the author wants to pass knowledge about error handling in a specific domain or language then I would prefer as a reader to have a different chapter that outlines all the dominant paradigms of error handling and how this affects the rest of the chapters.
I don't think error handling should be in the example if it obscures the logic. But some error handling is just the idiom of doing some things, and in theese case include it.
Also if pointing out that error handling needs to be added. For the love of deity also point out what errors needs to be handled.
This is the most frustrating part of reading some examples. If you don't know what you are doing (which we have to assume of the reader of the example...) you don't know what errors to look for either. Which turns the "add error handling" suggestion into "this example is useless".
One approach I've seen, notably in Advanced Programming in the UNIX Environment and UNIX Network Programming is to wrap calls with error checking code and then use the wrappers in the example code. For instance:
ssiz_t Recv(...)
{
ssize_t result;
result = recv(...);
/* error checking in full */
}
then, in calling code:
Recv(...);
That way you get to show error handling while allowing the flow of calling code to be clear and concise.
No, unless the purpose of the example is to demonstrate an aspect of exception handling. This is a pet peeve of mine -- many examples try to demonstrate best practices and end up obscuring and complicating the example. I see this all the time in code examples that start by defining a bunch of interfaces and inheritance chains that aren't necessary for the example. A prime example of over complicating was a hands-on lab I did at TechEd last year. The lab was on Linq, but the sample code I was directed to write created a multi-tier application for no purpose.
Examples should start with the simplest possible code that demonstrates the point, then progress into real-world usage and best practices.
As an aside, when I've asked for code samples from job candidates almost all of them are careful to demonstrate their knowledge of exception handling:
public void DoSomethingCool()
{
try
{
// do something cool
}
catch (Exception ex)
{
throw ex;
}
}
I've received hundreds of lines of code with every method like this. I've started to award bonus points for those that use throw; instead of throw ex;
Sample code need not include error handling but it should otherwise demonstrate proper secure coding techniques. Many web code snippets violate the OWASP Top ten.

Is it possible for the Vb.Net compiler to switch on an "Unreachable code" warning?

I've been mostly working with VB.Net for over a year and just noticed this
Am I going insane, or does VB.Net NOT have an "Unreachable code" warning?
The following compiles quite happily with nary a warning or error, even though there is a return between the two writeline calls.
Sub Main()
Console.WriteLine("Hello World")
Return
Console.WriteLine("Unreachable code, will never run")
End Sub
Am I missing something? Is there some way to switch this on that I can't find.
If not, is there a good reason for its omission? (i.e. or am I right in thinking this is a woeful state of affairs)
Forgive the air of rant about this question, it's not a rant, I would like an answer.
Thanks
I've raised this on MS Connect, as bug# 428529
Update
I received the following from the VB Teams program manager
Thanks for taking the time to report
this issue. The compiler has limited
support for this scenario, and as you
point out we don't have warnings for
unreachable code. There are some
scenarios that our flow analysis
algorithm does handle, such as the
following:
Sub Main()
Dim x As Integer
Return
x = 4
End Sub
In this case you'll get a warning that
x has never been assigned. For the
case you mentioned however we'll have
to look at implementing that for a
future release.
My guess is that it's an oversight in the compiler. Flow control is a very difficult problem to get correct in any language, but especially in a language like VB which has so many different flow control mechanisms. For instance,
Exceptions
Goto
On Error (Resume, Goto, etc ...)
Exit calls
If you feel strongly about this issue, please file a bug on Connect. We do take bugs filed via Connect very seriously and do our best to fix as many as possible.
They mention this in the following post:
https://stackoverflow.com/questions/210187/usage-statistics-c-versus-vb-net
See the last post.
I guess you could use FXCop to check your code instead or get a copy of Resharper from:
http://www.jetbrains.com/resharper/
I'd like to address Jared's answer.
Most of the issues he brings up are not problematic for data flow analysis.
The one exception is "On Error / Resume". They mess up data flow analysis pretty bad.
However, it's a pretty simple problem to mitigate:
If more than one "On Error" statement is used in a method, or the "Resume next" statement is used, you can just turn off data flow analysis and report a generic warning. A good one might be something like "On Error / Resume are deprecated, use exceptions instead." :)
In the common case of one only "On Error" statement and no "resume" statement, you can pretty much do normal data flow analysis, and should get reasonable results from it.
The big problem is with the way the existing DFA code is implemented. It doesn't use a control flow graph, and so changing it ends up being really expensive. I think if you want to address these kinds of issues you really need rip out the existing DFA code and replace it with something that uses a control flow graph.
AFAIK, you are correct that VB.NET does not give you a warning. C# does though.