I've read a lot of questions about multiple asserts in tests. Some are against it and some think it's OK. But I'm starting to wonder how I should do it with longer tests that have many steps.
For example this test with an Android device:
Start wifi
Install app
Uninstall app
Stop wifi
run test a couple of times
As I want to run it multiple times and always in this order it has to be a single test(?). So then I'm forced to do four asserts on the way:
Check that wifi is on.
Check that the app got installed.
Check that the app got uninstalled.
Check that wifi is off.
test is OK
Is this wrong or ugly? I don't see how I could get away from it without splitting up the test and as I see it as a single test case it also seems wrong.
From what I understand from the description: yes, this is wrong because of this part
always in this order
A good unit test is isolated (not dependent on other tests) and its results are not dependent on a particular order of execution. This is important because many frameworks simply have no guarantee to the order of execution.
I think you can split that test up in multiple tests. Keep in mind that in order to test something you might have to change the state prior to it (which is what you do with starting/stopping WIFI) so this is something hard to overcome.
This could be your layout of tests:
StartWifi
StopWifi
InstallApp_WithWifiStarted_InstallsSuccesfully
InstallApp_WithoutWifiStarted_AbortsInstallation
and continue like this for uninstall (I'm not sure what the requirements for that are).
With these tests you will now have knowledge of the following:
The wifi service can be started
The wifi service can be stopped
Installing the app with wifi works
Installing the app without wifi doesn't work
Whereas with your single test you could only deduce from a failure that something went wrong throughout the line but it's unclear where. The problem could have been located at
Starting wifi
Installing app
Uninstalling app
Stopping wifi
With separate, smaller tests you can rule out the ones that aren't applicable because they work themselves.
[At this point I notice you changed the tag from unit-testing to integration-testing]
It's important to note though that what you do isn't bad per sé: larger units are good to test as well although, as you indicate yourself, this is where you're getting close to integration testing.
It's important that you use unit-testing and integration-testing as a complementary testing method: by having these smaller unit tests and your bigger integration test, you can verify that the smaller parts work and that the combination of them works.
Conclusion: yes, having several asserts in your test is okay but make sure you also have smaller tests to test the independent units.
Yes, it's fine to use multiple asserts in a single test. Your test is an integration test and it looks like an acceptance test, and it is normal for those (which exercise a big part of the system) to have many assertions. There should only be one block of assertions, however.
To illustrate that, here are the four tests I think you need to test the functionality you're testing (considering only happy paths):
Test that the wifi can be turned on.
Turn the wifi on.
Assert that the wifi is on.
Turn the wifi off.
Test that the wifi can be turned off:
Turn the wifi on.
Turn the wifi off.
Assert that the wifi is off.
Test that the application can be installed:
Turn the wifi on.
Install the application
Assert that the application is installed.
Turn the wifi off.
Uninstall the application (if you need to do that to clean up).
Test that the application can be uninstalled:
Turn the wifi on.
Install the application.
Uninstall the application.
Assert that the application is uninstalled.
Turn the wifi off.
Each test tests only one action. It might take multiple language-level assertions to test that that action did everything it was supposed to; that's fine. The point is that there's only one block of assertions, and it's at the end of the test (not counting cleanup steps). Tests that need setup code don't need to assert anything about whether that setup code succeeded; that was already done in another test. Likewise, actions that are used in cleanup steps (the steps which follow the assertions) are tested in one place and don't need to be tested again when they're used for cleanup. Each action is tested in one place. The result is that you only need to read one test to find out how a piece of functionality should behave, and you're more likely to need to change only one test if the way that functionality should behave changes.
Related
I'm just curious if there's any known unwanted effect of this flag on automation, or if it can make my tests less valid.
I'm currently running tests with this flag and it doesn't seem to hurt anything. Is it just overlooked?
https://peter.sh/experiments/chromium-command-line-switches/#browser-test
https://github.com/GoogleChrome/puppeteer/blob/master/lib/Launcher.js#L38
The --browser-test activates an internal test used by the Chromium developers regarding canvas repaints.
Some older code in the repository, gives this hint
Tells Content Shell that it's running as a content_browsertest.
And this issue in the Chromium repository contains more information:
We need a test that checks canvas capture happens for N times when there are N repaints. This test is not appropriate for webkit layout tests as it is slow and there are mock streams involved.
Looks like they added a special flag for this test.
Therefore, you should not activate this flag as this test is about internal browser tests by the developer team not about testing websites.
Say there are 1-10 user stories. All tested okay. -> to Production. Then comes the CR with 5 more user stories. All then tested okay. -> to production.
Then comes 5 more user stories. Tested okay. -> To production.. now here a user story or two from first 1- 10 breaks down. Obviously testers will have to carry the blame for the same.
Developers have direct access to the QA environments' build path. any developer can go put the code file there. just a simple folder structure.
How do we fix this and keep 'our' hands clean?
Also Please note that we do ad-hoc testing due to the stringent timelines.
The situation when something new breaks down something old is rather common. I cannot see what is the problem. QA environment is perfectly good for catching up such a regression.
What i can suggest is:
1. Having Development / QA / Production environments
And try to set up the proper process of if sth new has been coded up and developer-tested it can go to 'QA'. And only when the new stuff has been QA-tested it can go to 'Production';
2. Continuous Build Integration
It's also nice to have the key features covered with the unit tests or (and) to have a suite of automated tests. One button-click can show you the general state of your app and even whose check-in has failed the build.
3. Regression testing
Ensure you have a profound Regression suite. These are run mainly to avoid such problems and verify that no critical issues leak into the production.
Hope this helps a little.
Our rails development team tries to follow Continuous Integration. We have decided to adopt a policy of only committing features whose tests pass. Is that a good way to go on? Should I delay integrating with other one's features until my tests pass(Even if the partial part of the feature works ok)? Thanks in advance
The tests should pass--if you're running a CI server it'll just spam people with emails until they do. Without a CI server everyone else will have to figure out if those tests are "supposed" to fail. Boo.
Another option is to only check in tests for actually-written features; if you're using tests as an executable specification they wouldn't all pass until the entire app was done and nobody would be able to check anything in ever.
You may also be able to mark tests as "pending" or indicate they should be skipped, but remembering to un–pend/-skip them is often problematic.
The tests SHOULD PASS that's the reason why you are writing them in the first place, if for some reason one or more tests do not pass, it indicates that something went wrong (obviously) and you and your team should be working on the solution.
If the code were committed with test failures, spam mails blaming the programmer who did it, this way the next time he will pay more attention before committing code
I have heard one way to avoid committing code with test failures but I have not personally tested, it involves to have two repositories (it could be a branch), the theory behind is:
The developers commits will target a branch, the purpose of this branch is just to guarantee that all tests pass, you should configure your CI server to build and run tests from this branch
When all the tests pass in the branch, a merge should be done to the trunk, since everyone should be working on this branch the merge should be transparent and automatic
I repeat I have not tested this approach and in my opinion it involves more problems than it solves
Another alternative could be to add a hook to the commit event in your VCS and force to run all tests but this could be time consuming just to perform a single commit
As additional info you could check this response
https://stackoverflow.com/a/7110774/1268570
I would wait personally to the test passes before I intergrate other features.
Where i am working we have the following issue:
Our current test procedure is that our business analyst test the release based on their specifications/tests. If it passes these tests it is given to the quality dept where they test the new release and the entire system to check if something else was broken.
Just to mention that we outsource our development. Unfortunately the release given to us is rarely tested by the developers and thats "the relationship" we have with them these last 7 years....
As a result if the patch/release fails the tests at the functionality testing level or at the quality level with each patch given we need to test the whole thing again not just the release.
Is there a way we can prevent this from happening?
You have two options:
Separate the code into independent modules so that a patch/change in one module only means you have to re-test that one module. However, due to dependencies this is effective only to a very limited degree.
Introduce automated tests so that re-testing is not as expensive. It takes some more work at fist, but will definitely pay off in your scenario. You don't have to do unit test or TDD - integration tests based on capture-replay tools are often easier to introduce in your scenario (established project with manual testing process).
Implement a continuous testing framework that you and the developers can access. Someething like CruiseControl.Net and NUnit to automate the functional tests.
Given access, they'll be able to see nightly tests on the build. Heck, they don't even need to test it themselves, your tests will be being run every night (or regularly), and they'll know straight away what faults they've caused, or fixed, if any.
Define a 'Quality SLA' - namely that all unit tests must pass, all new code must have a certain level of coverage, all new code must have a certain score in some static analysis checker.
Of course anything like this can be gamed, so have regular post release debriefs where you discuss areas of concern and put in place contingency to avoid it in future.
Implement GO server with Dashboard and handle with GO Agent GUI at your end.
http://www.thoughtworks-studios.com/forms/form/go/downloadlink text
Are there specific techniques to consider when refactoring the non-regression tests? The code is usually pretty simple, but it's obviously not included into the safety net of a test suite...
When building a non-regression test, I first ensure that it really exhibits the issue that I want to correct, of course. But if I come back later to this test because I want to refactor it (e.g. I just added another very similar test), I usually can't put the code-under-test back in a state where it was exhibiting the first issue. So I can't be sure that the test, once refactored, is still exercising the same paths in the code.
Are there specific techniques to deal with this issue, except being extra careful?
It's not a big problem. The tests test the code, and the code tests the tests. Although it's possible to make a clumsy mistake that causes the test to start passing under all circumstances, it's not likely. You'll be running the tests again and again, so the tests and the code they test gets a lot of exercise, and when things change for the worse, tests generally start failing.
Of course, be careful; of course, run the tests immediately before and after refactoring. If you're uncomfortable about your refactoring, do it in a way that allows you to see the test working (passing and failing). Find a reliable way to fail each test before the refactoring, and write it down. Get to green - all tests passing - then refactor the test. Run the tests; still green? Good. (If not, of course, get green, perhaps by starting over). Perform the changes that made the original unrefactored tests fail. Red? Same failure as before? Then reinstate the working code, and check for green again. Check it in and move onto your next task.
Try to include not only positive cases in your automated test, but also negative cases (and a proper handler for them).
Also, you can try to run your refactored automated test with breakpoints and supervise through the debugger that it keeps on exercising all the paths you intended it to exercise.