Is it a good practice to write a EXPECT(something) inside a test double (e.g. spy or mock) method? To ensure the test double is used in a specific way for testing?
If not, what would be a preferred solution?
If you would write a true Mock (as per definition from xUnit Test Patterns) this is exactly what defines this kind of test double. It is set up with the expectations how it will be called and therefore also includes the assertions. That's also how mocking frameworks produce mock objects under the hood. See also the definition from xUnit Test Patterns:
How do we implement Behavior Verification for indirect outputs of the SUT?
How can we verify logic independently when it depends on indirect inputs from other software components?
Replace an object the system under test (SUT) depends on with a test-specific object that verifies it is being used correctly by the SUT.
Here, indirect outputs means that you don't want to verify that the method under test returns some value but that there is something happening inside the method being tested that is behaviour relevant to callers of the method. For instance, that while executing some method the correct behaviour lead to an expected important action. Like sending an email or sending a message somewhere. The mock would be the doubled dependency that also verifies itself that this really happened, i.e. that the method under test really called the method of the dependency with the expected parameter(s).
A spy on the other hand shall just record things of interest that happened to the doubled dependency. Interrogating the spy about what happened (and sometimes also how often) and then judging if that was correct by asserting on the expected events is the responsibility of the test itself. So a mock is always also a spy with the addition of the assertion (expectation) logic. See also Uncle Bobs blog The Little Mocker for a great explanation of the different types of test doubles.
TL;DR
Yes, the mock includes the expectations (assertion) itself, the spy just records what happened and lets the test itself asks the spy and asserts on the expected events.
Mocking frameworks also implement mocks like explained above as they all follow the specified xunit framework.
mock.Verify(p => p.Send(It.IsAny<string>()));
If you look at the above Moq example (C#), you see that the mock object itself is configured to in the end perform the expected verification. The framework makes sure that the mock's verification methods are executed. A hand-written would be setup and than you would call the verification method on the mock object yourself.
Generally, you want to put all EXPECT statements inside individual tests to make your code readable.
If you want to enforce certain things on your test stub/spy, it is probably better to use exceptions or static asserts because your test is usually using them as a black box, and it uses them in an unintended way, your code will either not get compiled, or it will throw and give you the full stack trace which also will cause your test to fail (so you can catch the misuse).
For mocks, however, you have full control over the use and you can be very specific about how they are called and used inside each test. For example in Google test, using GMock matchers, you can say something like:
EXPECT_CALL(turtle, Forward(Ge(100)));
which means expect Forward to be called on the mock object turtle with a parameter equal or greater than 100. Any other value will cause the test to fail.
See this video for more examples on GMock matchers.
It is also very common to check general things in a test fixture (e.g. in Setup or TearDown). For example, this sample from google test enforces each test to finish in a certain amount of time, and the EXPECT statement is in teardown rather than each individual test.
I've been a professional software engineer for about a year now, having graduated with a CS degree. I've known about assertions for a while in C++ and C, but had no idea they existed in C# and .NET at all until recently.
Our production code contains no asserts whatsoever and my question is this...
Should I begin using Asserts in our production code? And if so, When is its use most appropriate? Would it make more sense to do
Debug.Assert(val != null, "message");
or
if ( val == null )
throw new exception("message");
In Debugging Microsoft .NET 2.0 Applications John Robbins has a big section on assertions. His main points are:
Assert liberally. You can never have too many assertions.
Assertions don't replace exceptions. Exceptions cover the things your code demands; assertions cover the things it assumes.
A well-written assertion can tell you not just what happened and where (like an exception), but why.
An exception message can often be cryptic, requiring you to work backwards through the code to recreate the context that caused the error. An assertion can preserve the program's state at the time the error occurred.
Assertions double as documentation, telling other developers what implied assumptions your code depends on.
The dialog that appears when an assertion fails lets you attach a debugger to the process, so you can poke around the stack as if you had put a breakpoint there.
PS: If you liked Code Complete, I recommend following it up with this book. I bought it to learn about using WinDBG and dump files, but the first half is packed with tips to help avoid bugs in the first place.
Put Debug.Assert() everywhere in the code where you want have sanity checks to ensure invariants. When you compile a Release build (i.e., no DEBUG compiler constant), the calls to Debug.Assert() will be removed so they won't affect performance.
You should still throw exceptions before calling Debug.Assert(). The assert just makes sure that everything is as expected while you're still developing.
FWIW ... I find that my public methods tend to use the if () { throw; } pattern to ensure that the method is being called correctly. My private methods tend to use Debug.Assert().
The idea is that with my private methods, I'm the one under control, so if I start calling one of my own private methods with parameters that are incorrect, then I've broken my own assumption somewhere--I should have never gotten into that state. In production, these private asserts should ideally be unnecessary work since I am supposed to be keeping my internal state valid and consistent. Contrast with parameters given to public methods, which could be called by anyone at runtime: I still need to enforce parameter constraints there by throwing exceptions.
Additionally, my private methods can still throw exceptions if something doesn't work at runtime (network error, data access error, bad data retrieved from a third party service, etc.). My asserts are just there to make sure that I haven't broken my own internal assumptions about the state of the object.
From Code Complete (Wikipedia):
8 Defensive Programming
8.2 Assertions
An assertion is code that’s used during development—usually a routine
or macro—that allows a program to check itself as it runs. When an
assertion is true, that means everything is operating as expected.
When it’s false, that means it has detected an unexpected error in the
code. For example, if the system assumes that a customer-information
file will never have more than 50,000 records, the program might
contain an assertion that the number of records is lessthan or equal
to 50,000. As long as the number of records is less than or equal to
50,000, the assertion will be silent. If it encounters more than
50,000 records, however, it will loudly “assert” that there is an
error in the program.
Assertions are especially useful in large, complicated programs and
in high reliability programs. They enable programmers to more quickly
flush out mismatched interface assumptions, errors that creep in when
code is modified, and so on.
An assertion usually takes two arguments: a boolean expression that
describes the assumption that’s supposed to be true and a message to
display if it isn’t.
(…)
Normally, you don’t want users to see assertion messages in
production code; assertions are primarily for use during development
and maintenance. Assertions are normally compiled into the code at
development time and compiled out of the code for production. During
development, assertions flush out contradictory assumptions,
unexpected conditions, bad values passed to routines, and so on.
During production, they are compiled out of the code so that the
assertions don’t degrade system performance.
Use asserts to check developer assumptions and exceptions to check environmental assumptions.
If I were you I would do:
Debug.Assert(val != null);
if ( val == null )
throw new exception();
Or to avoid repeated condition check
if ( val == null )
{
Debug.Assert(false,"breakpoint if val== null");
throw new exception();
}
If you want Asserts in your production code (i.e. Release builds), you can use Trace.Assert instead of Debug.Assert.
This of course adds overhead to your production executable.
Also if your application is running in user-interface mode, the Assertion dialog will be displayed by default, which may be a bit disconcerting for your users.
You can override this behaviour by removing the DefaultTraceListener: look at the documentation for Trace.Listeners in MSDN.
In summary,
Use Debug.Assert liberally to help catch bugs in Debug builds.
If you use Trace.Assert in user-interface mode, you probably want to remove the DefaultTraceListener to avoid disconcerting users.
If the condition you're testing is something your app can't handle, you're probably better off throwing an exception, to ensure execution doesn't continue. Be aware that a user can choose to ignore an assertion.
Asserts are used to catch programmer (your) error, not user error. They should be used only when there is no chance a user could cause the assert to fire. If you're writing an API, for example, asserts should not be used to check that an argument is not null in any method an API user could call. But it could be used in a private method not exposed as part of your API to assert that YOUR code never passes a null argument when it isn't supposed to.
I usually favour exceptions over asserts when I'm not sure.
In Short
Asserts are used for guards and for checking Design by Contract constraints, i.e. to ensure that the state of your code, objects, variables and parameters is operating within the boundaries and limits of your intended design.
Asserts should be for Debug and non-Production builds only. Asserts are typically ignored by the compiler in Release builds.
Asserts can check for bugs / unexpected conditions which ARE in the control of your system
Asserts are NOT a mechanism for first-line validation of user input or business rules
Asserts should not be used to detect unexpected environmental conditions (which are outside the control of the code) e.g. out of memory, network failure, database failure, etc. Although rare, these conditions are to be expected (and your app code cannot fix issues like hardware failure or resource exhaustion). Typically, exceptions will be thrown - your application can then either take corrective action (e.g. retry a database or network operation, attempt to free up cached memory), or abort gracefully if the exception cannot be handled.
A failed Assertion should be fatal to your system - i.e. unlike an exception, do not try and catch or handle failed Asserts - your code is operating in unexpected territory. Stack Traces and crash dumps can be used to determine what went wrong.
Assertions have enormous benefit:
To assist in finding missing validation of user inputs, or upstream bugs in higher level code.
Asserts in the code base clearly convey the assumptions made in the code to the reader
Assert will be checked at runtime in Debug builds.
Once code has been exhaustively tested, rebuilding the code as Release will remove the performance overhead of verifying the assumption (but with the benefit that a later Debug build will always revert the checks, if needed).
... More Detail
Debug.Assert expresses a condition which has been assumed about state by the remainder of the code block within the control of the program. This can include the state of the provided parameters, state of members of a class instance, or that the return from a method call is in its contracted / designed range.
Typically, asserts should crash the thread / process / program with all necessary info (Stack Trace, Crash Dump, etc), as they indicate the presence of a bug or unconsidered condition which has not been designed for (i.e. do not try and catch or handle assertion failures), with one possible exception of when an assertion itself could cause more damage than the bug (e.g. Air Traffic Controllers wouldn't want a YSOD when an aircraft goes submarine, although it is moot whether a debug build should be deployed to production ...)
When should you use Asserts?
At any point in a system, or library API, or service where the inputs to a function or state of a class are assumed valid (e.g. when validation has already been done on user input in the presentation tier of a system, the business and data tier classes typically assume that null checks, range checks, string length checks etc on input have been already done).
Common Assert checks include where an invalid assumption would result in a null object dereference, a zero divisor, numerical or date arithmetic overflow, and general out of band / not designed for behaviour (e.g. if a 32 bit int was used to model a human's age, it would be prudent to Assert that the age is actually between 0 and 125 or so - values of -100 and 10^10 were not designed for).
.Net Code Contracts
In the .Net Stack, Code Contracts can be used in addition to, or as an alternative to using Debug.Assert. Code Contracts can further formalize state checking, and can assist in detecting violations of assumptions at ~compile time (or shortly thereafter, if run as a background check in an IDE).
Design by Contract (DBC) checks available include:
Contract.Requires - Contracted Preconditions
Contract.Ensures - Contracted PostConditions
Invariant - Expresses an assumption about the state of an object at all points in its lifespan.
Contract.Assumes - pacifies the static checker when a call to non-Contract decorated methods is made.
Mostly never in my book.
In the vast majority of occasions if you want to check if everything is sane then throw if it isn't.
What I dislike is the fact that it makes a debug build functionally different to a release build. If a debug assert fails but the functionality works in release then how does that make any sense? It's even better when the asserter has long left the company and no-one knows that part of the code. Then you have to kill some of your time exploring the issue to see if it is really a problem or not. If it is a problem then why isn't the person throwing in the first place?
To me this suggests by using Debug.Asserts you're deferring the problem to someone else, deal with the problem yourself. If something is supposed to be the case and it isn't then throw.
I guess there are possibly performance critical scenarios where you want to optimise away your asserts and they're useful there, however I am yet to encounter such a scenario.
According to the IDesign Standard, you should
Assert every assumption. On average, every fifth line is an assertion.
using System.Diagnostics;
object GetObject()
{...}
object someObject = GetObject();
Debug.Assert(someObject != null);
As a disclaimer I should mention I have not found it practical to implement this IRL. But this is their standard.
All asserts should be code that could be optimised to:
Debug.Assert(true);
Because it's checking something that you have already assumed is true. E.g.:
public static void ConsumeEnumeration<T>(this IEnumerable<T> source)
{
if(source != null)
using(var en = source.GetEnumerator())
RunThroughEnumerator(en);
}
public static T GetFirstAndConsume<T>(this IEnumerable<T> source)
{
if(source == null)
throw new ArgumentNullException("source");
using(var en = source.GetEnumerator())
{
if(!en.MoveNext())
throw new InvalidOperationException("Empty sequence");
T ret = en.Current;
RunThroughEnumerator(en);
return ret;
}
}
private static void RunThroughEnumerator<T>(IEnumerator<T> en)
{
Debug.Assert(en != null);
while(en.MoveNext());
}
In the above, there are three different approaches to null parameters. The first accepts it as allowable (it just does nothing). The second throws an exception for the calling code to handle (or not, resulting in an error message). The third assumes it can't possibly happen, and asserts that it is so.
In the first case, there's no problem.
In the second case, there's a problem with the calling code - it shouldn't have called GetFirstAndConsume with null, so it gets an exception back.
In the third case, there's a problem with this code, because it should already have been checked that en != null before it was ever called, so that it isn't true is a bug. Or in other words, it should be code that could theoretically be optimised to Debug.Assert(true), sicne en != null should always be true!
Use assertions only in cases where you want the check removed for release builds. Remember, your assertions will not fire if you don't compile in debug mode.
Given your check-for-null example, if this is in an internal-only API, I might use an assertion. If it's in a public API, I would definitely use the explicit check and throw.
I thought I would add four more cases, where Debug.Assert can be the right choice.
1) Something I have not seen mentioned here is the additional conceptual coverage Asserts can provide during automated testing. As a simple example:
When some higher-level caller is modified by an author who believes they have expanded the scope of the code to handle additional scenarios, ideally (!) they will write unit tests to cover this new condition. It may then be that the fully integrated code appears to work fine.
However, actually a subtle flaw has been introduced, but not detected in test results. The callee has become non-deterministic in this case, and only happens to provide the expected result. Or perhaps it has yielded a rounding error that was unnoticed. Or caused an error that was offset equally elsewhere. Or granted not only the access requested but additional privileges that should not be granted. Etc.
At this point, the Debug.Assert() statements contained in the callee coupled with the new case (or edge case) driven in by unit tests can provide invaluable notification during test that the original author's assumptions have been invalidated, and the code should not be released without additional review. Asserts with unit tests are the perfect partners.
2) Additionally, some tests are simple to write, but high-cost and unnecessary given the initial assumptions. For example:
If an object can only be accessed from a certain secured entry point, should an additional query be made to a network rights database from every object method to ensure the caller has permissions? Surely not. Perhaps the ideal solution includes caching or some other expansion of features, but the design does not require it. A Debug.Assert() will immediately show when the object has been attached to an insecure entry point.
3) Next, in some cases your product may have no helpful diagnostic interaction for all or part of its operations when deployed in release mode. For example:
Suppose it is an embedded real-time device. Throwing exceptions and restarting when it encounters a malformed packet is counter-productive. Instead the device may benefit from best-effort operation, even to the point of rendering noise in its output. It also may not have a human interface, logging device, or even be physically accessible by human at all when deployed in release mode, and awareness of errors is best provided by assessing the same output. In this case, liberal Assertions and thorough pre-release testing are more valuable than exceptions.
4) Lastly, some tests are unneccessary only because the callee is perceived as extremely reliable. In most cases, the more reusable code is, the more effort has been put into making it reliable. Therefore it is common to Exception for unexpected parameters from callers, but Assert for unexpected results from callees. For example:
If a core String.Find operation states it will return a -1 when the search criteria is not found, you may be able to safely perform one operation rather than three. However, if it actually returned -2, you may have no reasonable course of action. It would be unhelpful to replace the simpler calculation with one that tests separately for a -1 value, and unreasonable in most release environments to litter your code with tests ensuring core libraries are operating as expected. In this case Asserts are ideal.
Quote Taken from The Pragmatic Programmer: From Journeyman to Master
Leave Assertions Turned On
There is a common misunderstanding about assertions, promulgated by
the people who write compilers and language environments. It goes
something like this:
Assertions add some overhead to code. Because they check for things
that should never happen, they'll get triggered only by a bug in the
code. Once the code has been tested and shipped, they are no longer
needed, and should be turned off to make the code run faster.
Assertions are a debugging facility.
There are two patently wrong assumptions here. First, they assume that
testing finds all the bugs. In reality, for any complex program you
are unlikely to test even a miniscule percentage of the permutations
your code will be put through (see Ruthless Testing).
Second, the optimists are forgetting that your program runs in a
dangerous world. During testing, rats probably won't gnaw through a
communications cable, someone playing a game won't exhaust memory, and
log files won't fill the hard drive. These things might happen when
your program runs in a production environment. Your first line of
defense is checking for any possible error, and your second is using
assertions to try to detect those you've missed.
Turning off assertions when you deliver a program to production is
like crossing a high wire without a net because you once made it
across in practice. There's dramatic value, but it's hard to get life
insurance.
Even if you do have performance issues, turn off only those
assertions that really hit you.
You should always use the second approach (throwing exceptions).
Also if you're in production (and have a release-build), it's better to throw an exception (and let the app crash in the worst-case) than working with invalid values and maybe destroy your customer's data (which may cost thousand of dollars).
You should use Debug.Assert to test for logical errors in your programs. The complier can only inform you of syntax errors. So you should definetely use Assert statements to test for logical errors. Like say testing a program that sells cars that only BMWs that are blue should get a 15% discount. The complier could tell you nothing about if your program is logically correct in performing this but an assert statement could.
I've read the answers here and I thought I should add an important distinction. There are two very different ways in which asserts are used. One is as a temporary developer shortcut for "This shouldn't really happen so if it does let me know so I can decide what to do", sort of like a conditional breakpoint, for cases in which your program is able to continue. The other, is a as a way to put assumptions about valid program states in your code.
In the first case, the assertions don't even need to be in the final code. You should use Debug.Assert during development and you can remove them if/when no longer needed. If you want to leave them or if you forget to remove them no problem, since they won't have any consequence in Release compilations.
But in the second case, the assertions are part of the code. They, well, assert, that your assumptions are true, and also document them. In that case, you really want to leave them in the code. If the program is in an invalid state it should not be allowed to continue. If you couldn't afford the performance hit you wouldn't be using C#. On one hand it might be useful to be able to attach a debugger if it happens. On the other, you don't want the stack trace popping up on your users and perhaps more important you don't want them to be able to ignore it. Besides, if it's in a service it will always be ignored. Therefore in production the correct behavior would be to throw an Exception, and use the normal exception handling of your program, which might show the user a nice message and log the details.
Trace.Assert has the perfect way to achieve this. It won't be removed in production, and can be configured with different listeners using app.config.
So for development the default handler is fine, and for production you can create a simple TraceListener like below which throws an exception and activate it in the production config file.
using System.Diagnostics;
public class ExceptionTraceListener : DefaultTraceListener
{
[DebuggerStepThrough]
public override void Fail(string message, string detailMessage)
{
throw new AssertException(message);
}
}
public class AssertException : Exception
{
public AssertException(string message) : base(message) { }
}
And in the production config file:
<system.diagnostics>
<trace>
<listeners>
<remove name="Default"/>
<add name="ExceptionListener" type="Namespace.ExceptionTraceListener,AssemblyName"/>
</listeners>
</trace>
</system.diagnostics>
I don't know how it is in C# and .NET, but in C will assert() only work if compiled with -DDEBUG - the enduser will never see an assert() if it's compiled without. It's for developer only. I use it really often, it's sometimes easier to track bugs.
I would not use them in production code. Throw exceptions, catch and log.
Also need to be careful in asp.net, as an assert can show up on the console and freeze the request(s).