currently, I am looking deeper into testing techniques, even though I am not sure if I still reside in the unittest-land or left it into the land of integration tests already.
Let me elaborate a bit, Given two components A and B and A uses B, then we have a certain "upwards-contract" for B and a certain "downwards-contract" for A. Basically this means: If A uses B correctly and B behaves correctly, then both contracts will be fulfilled and things will work correctly.
I think mocks are a way to guarantee a subset of an upwards-contract that is required for a given testcase. For example, a database connection might have the upwards contract to retrieve data records if they have been inserted earlier. A database connection mock guarantees to return certain records, without requiring their insertion into the database.
However, I am currently wondering if there is a way to verify the downwards-contract as well. Given the example of the database connection, the downwards-contract might be: You must connect to the database and ensure the connection exists and works and enter correct SQL-Queries.
Does anyone do something like this? Is this worth the work for more complicated contracts? (For example, the database connection might require an SQL-parser in order to completely verify calls to the database layer)
Greetings, tetha
This is really the difference between mocks and stubs - mocks verify exactly that (or at least can do so - you can use mocks as stubs with most frameworks). Essentially, mocks allows you to do protocol testing rather than just "if you call X I'll give you Y". Every mocking framework I've used allows you to easily verify things like "all these calls were made" and "these calls happened in a particular order".
The more protocol you enforce between components, the more brittle the tests will be - sometimes that's entirely appropriate (e.g. "you must authenticate before you perform any other operations") but it's easy to end up with tests which have to be changed every time you change the implementation, even in a reasonable way.
Does anyone do this?
Yes, I sometimes use mocks to verify "downward-contracts".
E.g. You can use the DB mock to check, if the correct
credentials where used for the login.
Especially, if You have interfaces to other subsystems,
then You can mock them up and let the mockup check for usage
violations.
E.g. if a subsystem requires an initialization
call or some kind of registration, then Your mock-up of the
subsystem interface could enforce this, too.
Is it worth the work?
It depends on how deep You want Your tests to be, let me give You some examples of different "deepness":
If You want to check the proper sequence of calls to an interface, then a simple state machine may be sufficient.
If You want to verify the proper usage of an interface language (SQL in Your example), You have to use a parser.
If You want to verify that it actually works with the real subsystem, then make an integration test (cannot be done with mockups).
Conclusion:
If appropriate, it should be done. However, You cannot expect a mockup to find each and every wrong use of the interface. E.g. a database mockup hardly detects, if there will be a deadlock caused by two concurrent transactions.
Related
In the scala and scalajs library Diode, I have used but not entirely understood the PotAction class and only recently discovered the AsyncAction class, both of which seem to be favored in situations involving, well, asynchronous requests. While I understand that, I don't entirely understand the design decisions and the naming choices, which seem to suggest a more narrow use case.
Specifically, both AsyncAction and PotAction require an initialModel and a next, as though both are modeling an asynchronous request for some kind of refreshable, updateable content rather than a command in the sense of CQRS. I have a somewhat-related question open regarding synchronous actions on form inputs by the way.
I have a few specific use cases in mind. I'd like to know a sketch (not asking for implementation, just the concept) of how you use something like PotAction in conjunction with any of:
Username/password authentication in a conventional flow
OpenAuth-style authentication with a third-party involved and a redirect
Token or cookie authentication behind the scenes
Server-side validation of form inputs
Submission of a command for a remote shell
All of these seem to be a bit different in nature to what I've seen using PotAction but I really want to use it because it has already been helpful when I am, say, rendering something based on the current state of the Pot.
Historically speaking, PotAction came first and then at a later time AsyncAction was generalized out of it (to support PotMap and PotVector), which may explain their relationship a bit. Both provide abstraction and state handling for processing async actions that retrieve remote data. So they were created for a very specific (and common) use case.
I wouldn't, however, use them for authentication as that is typically something you do even before your application is loaded, or any data requested from the server.
Form validation is usually a synchronous thing, you don't do it in the background while user is doing something else, so again Async/PotAction are not a very good match nor provide much added value.
Finally for the remote command use case PotAction might be a good fit, assuming you want to show the results of the command to the user when they are ready. Perhaps PotStream would be even better, depending on whether the command is producing a steady stream of data or just a single message.
In most cases you should use the various Pot structures for what they were meant for, that is, fetching and updating remote data, and maybe apply some of the ideas or internal models (such as the retry mechanism) to other request types.
All the Pot stuff was separated from Diode core into its own module to emphasize that they are just convenient helpers for working with Diode. Developers should feel free to create their own helpers (and contribute back to Diode!) for new use cases.
I would like your suggestion for my scenario:
I am implementing automated tests using bdd technique with Cucumber and Selenium WebDriver tools, what is currently happening is: a lot of scenarios depend of data of each other, so right now I am storing these data in the class I define the steps, so I can use in other scenarios.
But, as the application grows, and the more scenario I get, the more mess my application gets.
Do you have any design pattern, or solution I could use in this case?
As you say, scenarios that depends on data from other scenarios gets complicated and messy. The order of execution is important.
What would happen if you executed the scenarios in a random order? How would that affect you?
My approach would be to work hard on making each scenario independent of each other. If you have a flow like placing an order that is required for preparing a shipment that is required for creating an invoice and so in, then would I make sure sure that the state of the application was set correct before each scenario. That is, executing code that creates the desired state.
That was a complicated way to say that, in order to create an invoice I must first set the application state so that it has prepared a shipment. And possible other things as well.
I would work hard on setting the application to a known state before any scenario is executed. If that means cleaning the database, then I would do that. The goal would be to be able to execute each scenario in isolation.
The functionality in your system may build on each other. That doesn’t mean that the scenarios you use to check that your application still works should build on each other during their execution.
Not sure if this qualifies as a pattern, but it may be a direction to strive for.
Overall you want to be looking at a Separation of concerns and the Single Responsibility Principle.
At the Cucumber level have two 'layers' of responsibility, the Test Script (Feature File + Step Implementation) and model of the system under test. The Step Implementation maps straight onto the model. Its single purpose is binding feature steps to methods. The Model implementation is to model the state of the system under test which includes state persistence. The model should expose its interface in the declarative style over the imperative approach such that we see fooPage.login(); in preference to page.click('login');
On the Selenium WebDriver side of things use the Page Objects Model It is these reusable object that understand the semantics of representing page and would be a third layer.
Layers
- Test Script (Feature File + Java Steps)
- Model of SUT (which persists the state)
- Page Object Model -> WebDriver/Browser
As already pointed out, try to isolate test scenarios from each other regarding data.
Just a few approaches:
Either cleaning the database or restoring the original data before each test scenario is executed will make it; however, that can slow the tests down significantly. If the cleaning action takes around 10 seconds, that makes ~15 additional minutes for 100 tests, and ~3 hours for 1000 tests.
Alternatively, each test could generate and use its own data. The problem here is that many tests could really use the same data, in which case it makes little sense to create those data over and over again, let alone this is something that also takes time.
Yet another option is to tell between read-only tests and read-write tests. The former could make use of the default data, as they are not affected by data dependencies. The latter should deal with specific data to avoid running into conflicts with other test scenarios.
Still, step definitions within a test scenario are likely to depend on the state of the previous step definitions executed as part of that test scenario. So state management is still required somehow. You may need some helper Object in your Model.
Keep in mind that Steps classes are instantiated for every test scenario, along with the Objects created from them. Thus, private instance attributes won't work, unless all the steps used by a test scenario are implemented in the same Steps class. Otherwise think about static variables or try with a dependency injection framework.
I apologize if this has been answered before, but I'm having trouble finding a tool that fits my needs.
I have a few dozen test programs, but each one can be run with a large number of parameters. I need to be able to automatically run sweeps of many of the parameters across all or some of the test programs. I have my own set of tools for running an individual test, which I can't really change, but I'm looking for a tool that would manage the entire suite.
Thus far, I've used a home-grown script for this. The main problem I run across is that an individual test program might take 5-10 parameters, each with several values. Although it would be easy to write something that would just do a nested for loop and sweep over every parameter combination, the difficulty is that not every combination of parameters makes sense, and not every parameter makes sense for every test program. There is no general way (i.e., that works for all parameters) to codify what makes sense and what doesn't, so the solutions I've tried before involve enumerating each sensible case. Although the enumeration is done with a script, it still leads to a huge cross-product of test cases which is cumbersome to maintain. We also don't want to run the giant cross-product of cases every time, so I have other mechanisms to select subsets of it, which gets even more cumbersome to deal with.
I'm sure I'm not the first person to run into a problem like this. Are there any tools out there that could help with this kind of thing? Or even ideas for writing one?
Thanks.
Adding a clarification ---
For instance, if I have parameters A, B, and C that each represent a range of values from 1 to 10, I might have a restriction like: if A=3, then only odd values of B are relevant and C must be 7. The restrictions can generally be codified, but I haven't found a tool where I could specify something like that. As for a home-grown tool, I'd either have to enumerate the tuples of parameters (which is what I'm doing) or put or implement something quite sophisticated to be able to specify and understand constraints like that.
We rolled our own, we have a whole test infrastructure. It manages the tests, has a number of built in features for allowing the tests to log results, the logs are managed by the test infrastructure to go into a searchable database for all kinds of report generation.
Each test has a class/structure that has information about the test, name of test, author, and a variety of other tags. When running a test suite you can run everything or run everything with a certain tag. So if you want to only test SRAM you can easily run only tests tagged sram.
Our tests are all considered either pass or fail. pass/fail criteria is determined by the author of the individual test, but the infrastructure wants to see either pass or fail. You need to define what your possible results are, as simple as pass/fail or you might want to add pass and keep going, pass but stop testing, fail but keep going, and fail and stop testing. Stop testing meaning if there are 20 tests scheduled and test 5 fails then you stop you dont go on to 6.
You need a mechanism to order the tests which could be alphabetical but it might benefit from a priority scheme (must perform the power on test before performing a test that requires the power to be on). It may also benefit from a random ordering some tests may be passing due to dumb luck because a test before them made something work, remove that prior test and this test fails. or vice versa this test passes until it is preceeded by a specific test and those two dont get along in that order.
To shorten my answer I dont know of an existing infrastructure, but I have built my own and worked with home built ones that were tailored to our business/lab/process. You wont hit a home run the first time, dont expect to. but try to predict a managable set of rules for individual tests, how many types of pass/fail return values it can return. The types of filters you want to put in place. The type of logging you may wish to do and where you want to store that data. then create the infrastructure and the mandantory shell/frame for each test, then individual testers have to work within that shell. Our current infrastructure is in python which lent itself to this nicely, and we are not restricted to only python based tests we can use C or python and the target can run whatever languages/programs it can run. Abstraction layers are good, we use a simple read/write of an address to access the unit under test, and with that we can test against a simulation of the target or against real hardware when the hardware arrives. We can access the hardware through a serial debugger, or jtag or pcie, and the majority of the tests dont know or care because the are on the other side of the abstraction.
I am starting out with automated testing and I would like to test one of my data access methods. I am trying to test what the code does if the database returns no records.
Is this something that should be done in a unit test or an integration test?
Thanks
If your test code connects to an actual database and relies on the presence of certain data (or lack of data) in order for the test to pass, it's an integration test.
I ususally prefer to test something like this by mocking out the component that the "data access method" used to get the actual data, whether that's a JDBC connection or web service proxy or whatever else. With a mock, you say "when this method is called, return this" or "make sure that this method is called N times", and then you tell the class under test to use the mock component rather than the real component. This then is a "unit test", because you are testing how the class under test behaves, in a closed system where you've declared exactly how the other components will behave. You've isolated the class under test completely and can be sure that your test results won't be volatile and dependent on the state of another component.
Not sure what language/technology you are working with, but in the Java world, you can use JMock, EasyMock, etc for this purpose.
I think more time has been wasted arguing about what is a unit vs. what is an integration test than value has been added.
I don't care.
Let me put it a different way: If I were testing it, I'd see two ways to do it - fake out the database returning zero rows, or actually connect to a database that has no data for the select. I'd probably start testing with whatever was easiest to do and simplest to implement - if it ran fast enough for me to get meaningful feedback. Then I'd consider the other if I needed it to run faster or thought there would be some advantage.
For example, I'd probably start connecting to the actual test DB at my work. But if the software needed to work with many different databases - Oracle, PostGres, MySQL, SQL server and DB, or if the test DB at work was down for 'refreshes' a lot, I'd probably write the 'pure/unit' test that existed totally in isolation.
In my old age, I prefer to use the term 'developer-facing' vs. 'customer facing' more often, and do the kind of testing that makes more sense. I find using terms like "unit" extensively, then getting a definition-weenie about it leads to people doing things like mocking out the filesystem or mocking getters and setters - activity that I find unhelpful.
I believe this strongly; I've presented before google on it.
http://www.google.com/url?sa=t&source=web&oi=video_result&ct=res&cd=1&url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPHtEkkKXSiY&ei=9-wKSobjEpKANvHT_MEB&rct=j&q=heusser+GTAC+2007&usg=AFQjCNHOgFzsoVss50Qku1p011J4-UjhgQ
good luck! Let us know how it goes!
Do your test and let other people spend time with taxonomy.
My perspective is that you should categorize the test based on scope:
A unit test can be run standalone
without any external dependencies
(File IO, Network IO, Database,
External Web Services).
An integration test can touch external systems.
If the test requires a real database to run then call it an integration test and keep it separate from the unit tests. This is important because if you mix integration and unit tests than you make your code less maintainable.
A mixed bag of tests mean that new developers may need a whole heap of external dependencies in order to run the test suite. Imagine that you want to make a change to a piece of code that is related to the database but doesn't actually require the database to function, you're going to be frustrated if you need a database just to run the tests associated with the project.
If the external dependency is difficult to mock out (for example, in DotNet, if you are using Rhino Mocks and the external classes don't have interfaces) then create a thin wrapper class that touches the external system. Then mock out that wrapper in the unit tests. You shouldn't need a database to run this simple test so don't require one!
There are those (myself included) who have strict rules about what constitutes a unit test vs an integration test.
A test is not a unit test if:
It talks to the database
It communicates across the network
It touches the file system
It can’t run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing config files) to run it
Which may be one way to make a distinction between what a unit test will be doing for you using mocking for example, rather than any of the real resource providers - filesystem, db etc.
An integration test can be viewed as a test of very coupling of systems/application layers, so the fundamentals are tested in the unit and the system interoperability is the focus of an integration test.
Its still a grey area though because one can often pinpoint certain exceptions to these sorts of rules.
I think the important question is "What SHOULD I be doing?"
In this case I think you should be unit testing. Mock the code that talks to the DB and have it return a reliable result (no rows), this way your test checks what happens when there are no rows, and not what happens when the DB returns whatever is in the DB at the point you test.
Definitely unit test it!
[TestMethod]
public void ForgotMyPassword_SendsAnEmail_WhenValidUserIsPassed()
{
var userRepository = MockRepository.GenerateStub<IUserRepository>();
var notificationSender = MockRepository.GenerateStub<INotificationSender>();
userRepository.Stub(x => x.GetUserByEmailAddressAndPassword("me#home.com", "secret")).Return(new User { Id = 5, Name = "Peter Morris" });
new LoginController(userRepository, notificationSender).ResendPassword("me#home.com", "secret");
notificationSender.AssertWasCalled(x => x.Send(null),
options => options.Constraints(Text.StartsWith("Changed")));
}
I believe that it is possible to test that as a unit test, without a real database. Instead of using a real interface to the database, replace it with a mock/stub/fake object (better visualized PDF is here).
If writing it as a unit test proves to be too hard, and you are not able to refactor the code that testing it would be easy, then you better write it as an integration test. It will run slower, so you might not be able to run all the integration tests after ever code change (unlike unit tests which you can run hundreds and thousands per second), but as long as they are run regularly (for example as part of continous integration), they produce some value.
Most likely a unit test ... but there is a blurred line here. It really depends upon how much code is being executed - if it is contained to a library or class then its unit test, if it spans multiple components then it's more of an integration test.
I believe that should be done in a unit test. You aren't testing that it can connect to the database, or that you can call your stored procedures... you are testing the behavior of your code.
I could be wrong, but that's what I think unless someone gives me a reason to think otherwise.
that is a unit test, by definition: you are testing a single isolated element of the code on a specific path
I'm connecting to a simple, if idiosyncractic, external service.
I believe that my unit tests should not depend on the availability or implementation of that external service, so I intend to mock it out.
I need the mock to accept and return realistic messages and responses - otherwise my tests won't represent real states of affairs. For example, it has to throw the right kind of errors - and there are at least 7 different ways it can fail (between you and me it's not a very well designed external service). Therefore, at a bare minimum I have to have a hash of message/response pairs.
So instead of reducing contingency, mocking has reintroduced it somewhere else. In fact, as the saying goes, now I've got two problems: I've got to be sure that what's in my hash is a fair representation of how the external service behaves. But surely the canonical source of what response object X gives to message m is X itself. Anything else is risky and messy.
Have I taken a wrong turn? How can I eliminate this apparent circularity?
EDIT I've clarified what I think the problem is in the light of Justice's helpful comments.
Let me refer you to my two answers at another question about unit testing just to avoid repeating myself, first.
The thing I think that mocking gives you in this environment is that it's rigorously specifying what you think the behavior of that external interface is going to be. This means that you've got a controlled test (something tells me this external service changes every so often.) So, not only can you test and debug your code with a known "good" sequence of responses, but you have a documented set of examples of what you expect.
If I were in that situation, and depending on the real service, I'd be tempted to write a unit test or mock for the external service as well. That way, if you observe a fault in real operation, you can (1) run the test against your code using the mock for the external interface, and (2) test the external service against your expectations.
The point, though, is to have something for which you have real confidence and which you completely control.
Your mock intended for unit tests should not accurately represent the external service. You should pick pre-defined sets of input and output values for the mocked external service. They may or may not be what the external service would actually return (but they should be "kind of" real). The objective of your unit tests is to double-check that your objects behave correctly, given this set of inputs and outputs, for a certain meaning of "correctly."
Make sure that anything your class depends on has an interface (as opposed to just a concrete implementation), and then you can use JMock. This will simplify your testing immensely. You can do things like tell it to expect a method call X, return a particular value, or throw an exception. It's really a timesaver.