Is there a way knowing (at coding time) which exceptions to expect when executing python code?
I end up catching the base Exception class 90% of the time since I don't know which exception type might be thrown (reading the documentation doesn't always help, since many times an exception can be propagated from the deep. And many times the documentation is not updated or correct).
Is there some kind of tool to check this (like by reading the Python code and libs)?
I guess a solution could be only imprecise because of lack of static typing rules.
I'm not aware of some tool that checks exceptions, but you could come up with your own tool matching your needs (a good chance to play a little with static analysis).
As a first attempt, you could write a function that builds an AST, finds all Raise nodes, and then tries to figure out common patterns of raising exceptions (e. g. calling a constructor directly)
Let x be the following program:
x = '''\
if f(x):
raise IOError(errno.ENOENT, 'not found')
else:
e = g(x)
raise e
'''
Build the AST using the compiler package:
tree = compiler.parse(x)
Then define a Raise visitor class:
class RaiseVisitor(object):
def __init__(self):
self.nodes = []
def visitRaise(self, n):
self.nodes.append(n)
And walk the AST collecting Raise nodes:
v = RaiseVisitor()
compiler.walk(tree, v)
>>> print v.nodes
[
Raise(
CallFunc(
Name('IOError'),
[Getattr(Name('errno'), 'ENOENT'), Const('not found')],
None, None),
None, None),
Raise(Name('e'), None, None),
]
You may continue by resolving symbols using compiler symbol tables, analyzing data dependencies, etc. Or you may just deduce, that CallFunc(Name('IOError'), ...) "should definitely mean raising IOError", which is quite OK for quick practical results :)
You should only catch exceptions that you will handle.
Catching all exceptions by their concrete types is nonsense. You should catch specific exceptions you can and will handle. For other exceptions, you may write a generic catch that catches "base Exception", logs it (use str() function) and terminates your program (or does something else that's appropriate in a crashy situation).
If you really gonna handle all exceptions and are sure none of them are fatal (for example, if you're running the code in some kind of a sandboxed environment), then your approach of catching generic BaseException fits your aims.
You might be also interested in language exception reference, not a reference for the library you're using.
If the library reference is really poor and it doesn't re-throw its own exceptions when catching system ones, the only useful approach is to run tests (maybe add it to test suite, because if something is undocumented, it may change!). Delete a file crucial for your code and check what exception is being thrown. Supply too much data and check what error it yields.
You will have to run tests anyway, since, even if the method of getting the exceptions by source code existed, it wouldn't give you any idea how you should handle any of those. Maybe you should be showing error message "File needful.txt is not found!" when you catch IndexError? Only test can tell.
The correct tool to solve this problem is unittests. If you are having exceptions raised by real code that the unittests do not raise, then you need more unittests.
Consider this
def f(duck):
try:
duck.quack()
except ??? could be anything
duck can be any object
Obviously you can have an AttributeError if duck has no quack, a TypeError if duck has a quack but it is not callable. You have no idea what duck.quack() might raise though, maybe even a DuckError or something
Now supposing you have code like this
arr[i] = get_something_from_database()
If it raises an IndexError you don't know whether it has come from arr[i] or from deep inside the database function. usually it doesn't matter so much where the exception occurred, rather that something went wrong and what you wanted to happen didn't happen.
A handy technique is to catch and maybe reraise the exception like this
except Exception as e
#inspect e, decide what to do
raise
Noone explained so far, why you can't have a full, 100% correct list of exceptions, so I thought it's worth commenting on. One of the reasons is a first-class function. Let's say that you have a function like this:
def apl(f,arg):
return f(arg)
Now apl can raise any exception that f raises. While there are not many functions like that in the core library, anything that uses list comprehension with custom filters, map, reduce, etc. are affected.
The documentation and the source analysers are the only "serious" sources of information here. Just keep in mind what they cannot do.
I ran into this when using socket, I wanted to find out all the error conditions I would run in to (so rather than trying to create errors and figure out what socket does I just wanted a concise list). Ultimately I ended up grep'ing "/usr/lib64/python2.4/test/test_socket.py" for "raise":
$ grep raise test_socket.py
Any exceptions raised by the clients during their tests
raise TypeError, "test_func must be a callable function"
raise NotImplementedError, "clientSetUp must be implemented."
def raise_error(*args, **kwargs):
raise socket.error
def raise_herror(*args, **kwargs):
raise socket.herror
def raise_gaierror(*args, **kwargs):
raise socket.gaierror
self.failUnlessRaises(socket.error, raise_error,
self.failUnlessRaises(socket.error, raise_herror,
self.failUnlessRaises(socket.error, raise_gaierror,
raise socket.error
# Check that setting it to an invalid value raises ValueError
# Check that setting it to an invalid type raises TypeError
def raise_timeout(*args, **kwargs):
self.failUnlessRaises(socket.timeout, raise_timeout,
def raise_timeout(*args, **kwargs):
self.failUnlessRaises(socket.timeout, raise_timeout,
Which is a pretty concise list of errors. Now of course this only works on a case by case basis and depends on the tests being accurate (which they usually are). Otherwise you need to pretty much catch all exceptions, log them and dissect them and figure out how to handle them (which with unit testing wouldn't be to difficult).
There are two ways that I found informative. The first one, run the code in iPython, which will display the exception type.
n = 2
str = 'me '
str + 2
TypeError: unsupported operand type(s) for +: 'int' and 'str'
In the second way we settle for catching too much and improve on it over time. Include a try expression in your code and catch except Exception as err. Print sufficient data to know what exception was thrown. As exceptions are thrown improve your code by adding a more precise except clause. When you feel that you have caught all relevant exceptions remove the all inclusive one. A good thing to do anyway because it swallows programming errors.
try:
so something
except Exception as err:
print "Some message"
print err.__class__
print err
exit(1)
normally, you'd need to catch exception only around a few lines of code. You wouldn't want to put your whole main function into the try except clause. for every few line you always should now (or be able easily to check) what kind of exception might be raised.
docs have an exhaustive list of built-in exceptions. don't try to except those exception that you're not expecting, they might be handled/expected in the calling code.
edit: what might be thrown depends on obviously on what you're doing! accessing random element of a sequence: IndexError, random element of a dict: KeyError, etc.
Just try to run those few lines in IDLE and cause an exception. But unittest would be a better solution, naturally.
This is a copy and pasted answer I wrote for How to list all exceptions a function could raise in Python 3?, I hope that is allowed.
I needed to do something similar and found this post. I decided I
would write a little library to help.
Say hello to Deep-AST. It's very early alpha but it is pip
installable. It has all of the limitations mentioned in this post
and some additional ones but its already off to a really good start.
For example when parsing HTTPConnection.getresponse() from
http.client it parses 24489 AST Nodes. It finds 181 total raised
Exceptions (this includes duplicates) and 8 unique Exceptions were
raised. A working code example.
The biggest flaw is this it currently does work with a bare raise:
def foo():
try:
bar()
except TypeError:
raise
But I think this will be easy to solve and I plan on fixing it.
The library can handle more than just figuring out exceptions, what
about listing all Parent classes? It can handle that too!
I came across the sentence, "Compile time abstraction of runtime behaviour", what is compile time abstraction here? My guesses would be,
like in a language, trying to optimize/do things that can be done at compile time, and only leaving room for things that can be done only at the run time,
For Example.
int a;
a = 5;// 5 can be assigned to a only at compile time(unless it is a const), because the user might have created the program, where he gets the input from commandline, stdin,fin etc etc
where as, int a// can be done at compile time, since you know the type right away......
It seems that you are confused by "compile-time abstraction" in
Static type checking is a compile-time abstraction of the runtime behaviour of your program, ...
(quote from the paper "Static Typing Where Possible, Dynamic Typing When Needed" that you've linked to in your comment)
If the word "abstraction" was replaced by "approximation", would that make more sense to you?
Given an expression E of type T, we can say that T approximates at compile-time the sort of values computed at run-time (when evaluating E). For example, say you have an expression [2+2*3] of type [integer] -- you could say that "this expression will evaluate to an integer".
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I've heard some voices saying that checking for a returned null value from methods is bad design. I would like to hear some reasons for this.
pseudocode:
variable x = object.method()
if (x is null) do something
The rationale behind not returning null is that you do not have to check for it and hence your code does not need to follow a different path based on the return value. You might want to check out the Null Object Pattern which provides more information on this.
For example, if I were to define a method in Java that returned a Collection I would typically prefer to return an empty collection (i.e. Collections.emptyList()) rather than null as it means my client code is cleaner; e.g.
Collection<? extends Item> c = getItems(); // Will never return null.
for (Item item : c) { // Will not enter the loop if c is empty.
// Process item.
}
... which is cleaner than:
Collection<? extends Item> c = getItems(); // Could potentially return null.
// Two possible code paths now so harder to test.
if (c != null) {
for (Item item : c) {
// Process item.
}
}
Here's the reason.
In Clean Code by Robert Martin he writes that returning null is bad design when you can instead return, say, empty array. Since expected result is an array, why not? It'll enable you to iterate over result without any extra conditions. If it's an integer, maybe 0 will suffice, if it's a hash, empty hash. etc.
The premise is to not force calling code to immediately handle issues. Calling code may not want to concern itself with them. That's also why in many cases exceptions is better than nil.
Good uses of returning null:
If null is a valid functional result, for example: FindFirstObjectThatNeedsProcessing() can return null if not found and the caller should check accordingly.
Bad uses: Trying to replace or hide exceptional situations such as:
catch(...) and return null
API dependency initialization failed
Out of disk space
Invalid input parameters (programming error, inputs must be sanitized by the caller)
etc
In those cases throwing an exception is more adequate since:
A null return value provides no meaningful error info
The immediate caller most likely cannot handle the error condition
There is no guarantee that the caller is checking for null results
However, Exceptions should not be used to handle normal program operation conditions such as:
Invalid username/password (or any user-provided inputs)
Breaking loops or as non-local gotos
Yes, returning NULL is a terrible design, in object-oriented world. In a nutshell, NULL usage leads to:
ad-hoc error handling (instead of exceptions)
ambiguous semantic
slow instead of fast failing
computer thinking instead of object thinking
mutable and incomplete objects
Check this blog post for a detailed explanation: http://www.yegor256.com/2014/05/13/why-null-is-bad.html. More in my book Elegant Objects, Section 4.1.
Who says this is bad design?
Checking for nulls is a common practice, even encouraged, otherwise you run the risk of NullReferenceExceptions everywhere. Its better to handle the error gracefully than throw exceptions when you don't need to.
Based on what you've said so far, I think there's not enough information.
Returning null from a CreateWidget()method seems bad.
Returning null from a FindFooInBar() method seems fine.
Its inventor says it is a billion dollar mistake!
It depends on the language you're using. If you're in a language like C# where the idiomatic way of indicating the lack of a value is to return null, then returning null is a good design if you don't have a value. Alternatively, in languages such as Haskell which idiomatically use the Maybe monad for this case, then returning null would be a bad design (if it were even possible).
If you read all the answers it becomes clear the answer to this question depends on the kind of method.
Firstly, when something exceptional happens (IOproblem etc), logically exceptions are thrown. When exactly something is exceptional is probably something for a different topic..
Whenever a method is expected to possibly have no results there are two categories:
If it is possible to return a neutral value, do so.
Empty enumrables, strings etc are good examples
If such a neutral value does not exist, null should be returned.
As mentioned, the method is assumed to possibly have no result, so it is not exceptional, hence should not throw an exception. A neutral value is not possible (for example: 0 is not especially a neutral result, depending on the program)
Untill we have an official way to denote that a function can or cannot return null, I try to have a naming convention to denote so.
Just like you have the TrySomething() convention for methods that are expected to fail, I often name my methods SafeSomething() when the method returns a neutral result instead of null.
I'm not fully ok with the name yet, but couldn't come up with anything better. So I'm running with that for now.
I have a convention in this area that served me well
For single item queries:
Create... returns a new instance, or throws
Get... returns an expected existing instance, or throws
GetOrCreate... returns an existing instance, or new instance if none exists, or throws
Find... returns an existing instance, if it exists, or null
For collection queries:
Get... always returns a collection, which is empty if no matching[1] items are found
[1] given some criteria, explicit or implicit, given in the function name or as parameters.
Exceptions are for exceptional circumstances.
If your function is intended to find an attribute associated with a given object, and that object does has no such attribute, it may be appropriate to return null. If the object does not exist, throwing an exception may be more appropriate. If the function is meant to return a list of attributes, and there are none to return, returning an empty list makes sense - you're returning all zero attributes.
It's not necessarily a bad design - as with so many design decisions, it depends.
If the result of the method is something that would not have a good result in normal use, returning null is fine:
object x = GetObjectFromCache(); // return null if it's not in the cache
If there really should always be a non-null result, then it might be better to throw an exception:
try {
Controller c = GetController(); // the controller object is central to
// the application. If we don't get one,
// we're fubar
// it's likely that it's OK to not have the try/catch since you won't
// be able to really handle the problem here
}
catch /* ... */ {
}
It's fine to return null if doing so is meaningful in some way:
public String getEmployeeName(int id){ ..}
In a case like this it's meaningful to return null if the id doesn't correspond to an existing entity, as it allows you to distinguish the case where no match was found from a legitimate error.
People may think this is bad because it can be abused as a "special" return value that indicates an error condition, which is not so good, a bit like returning error codes from a function but confusing because the user has to check the return for null, instead of catching the appropriate exceptions, e.g.
public Integer getId(...){
try{ ... ; return id; }
catch(Exception e){ return null;}
}
For certain scenarios, you want to notice a failure as soon as it happens.
Checking against NULL and not asserting (for programmer errors) or throwing (for user or caller errors) in the failure case can mean that later crashes are harder to track down, because the original odd case wasn't found.
Moreover, ignoring errors can lead to security exploits. Perhaps the null-ness came from the fact that a buffer was overwritten or the like. Now, you are not crashing, which means the exploiter has a chance to execute in your code.
What alternatives do you see to returning null?
I see two cases:
findAnItem( id ). What should this do if the item is not found
In this case we could: Return Null or throw a (checked) exception (or maybe create an item and return it)
listItemsMatching (criteria) what should this return if nothing is found?
In this case we could return Null, return an empty list or throw an Exception.
I believe that return null may be less good than the alternatives becasue it requires the client to remember to check for null, programmers forget and code
x = find();
x.getField(); // bang null pointer exception
In Java, throwing a checked exception, RecordNotFoundException, allows the compiler to remind the client to deal with case.
I find that searches returning empty lists can be quite convenient - just populate the display with all the contents of the list, oh it's empty, the code "just works".
Make them call another method after the fact to figure out if the previous call was null. ;-) Hey, it was good enough for JDBC
Well, it sure depends of the purpose of the method ... Sometimes, a better choice would be to throw an exception. It all depends from case to case.
Sometimes, returning NULL is the right thing to do, but specifically when you're dealing with sequences of different sorts (arrays, lists, strings, what-have-you) it is probably better to return a zero-length sequence, as it leads to shorter and hopefully more understandable code, while not taking much more writing on API implementer's part.
The base idea behind this thread is to program defensively. That is, code against the unexpected.
There is an array of different replies:
Adamski suggests looking at Null Object Pattern, with that reply being up voted for that suggestion.
Michael Valenty also suggests a naming convention to tell the developer what may be expected.
ZeroConcept suggests a proper use of Exception, if that is the reason for the NULL.
And others.
If we make the "rule" that we always want to do defensive programming then we can see that these suggestions are valid.
But we have 2 development scenarios.
Classes "authored" by a developer: The Author
Classes "consumed" by another(maybe) developer: the Developer
Regardless of whether a class returns NULL for methods with a return value or not,
the Developer will need to test if the object is valid.
If the developer cannot do this, then that Class/method is not deterministic.
That is, if the "method call" to get the object does not do what it "advertises" (eg getEmployee) it has broken the contract.
As an author of a class, I always want to be as kind and defensive ( and deterministic) when creating a method.
So given that either NULL or the NULL OBJECT (eg if(employee as NullEmployee.ISVALID)) needs to be checked
and that may need to happen with a collection of Employees, then the null object approach is the better approach.
But I also like Michael Valenty's suggestion of naming the method that MUST return null eg getEmployeeOrNull.
An Author who throws an exception is removing the choice for the developer to test the object's validity,
which is very bad on a collection of objects, and forces the developer into exception handling
when branching their consuming code.
As a developer consuming the class, I hope the author gives me the ability to avoid or program for the null situation
that their class/methods may be faced with.
So as a developer I would program defensively against NULL from a method.
If the author has given me a contract that always returns a object (NULL OBJECT always does)
and that object has a method/property by which to test the validity of the object,
then I would use that method/property to continue using the object, else the object is not valid
and I cannot use it.
Bottom line is that the Author of the Class/Methods must provide mechanisms
that a Developer can use in their defensive programming. That is, a clearer intention of the method.
The Developer should always use defensive programming to test the validity of the objects returned
from another class/method.
regards
GregJF
Other options to this, are:
returning some value that indicates success or not (or type of an error), but if you just need boolean value that will indicate success / fail, returning null for failure, and an object for success wouldn't be less correct, then returning true/false and getting the object through parameter.
Other approach would to to use exception to indicates failures, but here - there are actually many more voices, that say this is a BAD practice (as using exceptions may be convenient but has many disadvantages).
So I personally don't see anything bad in returning null as indication that something went wrong, and checking it later (to actually know if you have succeeded or not). Also, blindly thinking that your method will not return NULL, and then base your code on it, may lead to other, sometimes hard to find, errors (although in most cases it will just crash your system :), as you will reference to 0x00000000 sooner or later).
Unintended null functions can arise during the development of a complex programs, and like dead code, such occurrences indicate serious flaws in program structures.
A null function or method is often used as the default behavior of a revectorable function or overrideable method in an object framework.
Null_function #wikipedia
If the code is something like:
command = get_something_to_do()
if command: # if not Null
command.execute()
If you have a dummy object whose execute() method does nothing, and you return that instead of Null in the appropriate cases, you don't have to check for the Null case and can instead just do:
get_something_to_do().execute()
So, here the issue is not between checking for NULL vs. an exception, but is instead between the caller having to handle special non-cases differently (in whatever way) or not.
For my use case I needed to return a Map from method and then looking for a specific key. But if I return an empty Map, then it will lead to NullPointerException and then it wont be much different returning null instead of an empty Map.
But from Java8 onward we could use Optional. The above is the very reason Optional concept was introduced.
G'day,
Returning NULL when you are unable to create a new object is standard practise for many APIs.
Why the hell it's bad design I have no idea.
Edit: This is true of languages where you don't have exceptions such as C where it has been the convention for many years.
HTH
'Avahappy,