I'd like to be able to set a condition from which to decide whether certain tests run in NUNIT.
For example if a global variable x = 1 then only run the tests from a certain class/assembly or only run the first test.
Is anything like this possible?
How would one go about doing this?
Thanks!!
I would recommend using Categories with your NUnit tests. This would allow you to run groups of them at a time, or all at once.
While Categories seem like the "more pure" way to do this, you could programmatically skip tests like this post asks: Programmatically skip an nunit test. It seems to me, this approach is what you're looking for though.
Related
Can you manually set outcome of test using XUnit? In one of my test I have to fulfill prerequisite and if it fails I need to set outcome of test to inconclusive. In NUnit you can set outcome by Assert.Inconclusive(), Assert.Fail(), etc. Am I able to do something similar with XUnit? Is there some best practice how to do this?
There is no out-of-the-box way of doing what you want in xUnit. See https://xunit.github.io/docs/comparisons.html for more information.
However Assert.Fail("This test is failing") can be replicated using Assert.True(false, "This test is failing").
The closest equivalent of Assert.Inconclusive in XUnit would be the use of the Skip attribute. But as far as I can see, there is no way to invoke this part way through a method, like you can with NUnit's Assert.Inconclusive.
Someone did write an Assert.Skip method for v1.9 here: https://github.com/xunit/xunit/blob/v1/samples/AssertExamples/DynamicSkipExample.cs, however it no longer compiles in v.2.2.0. Actually the xUnit authors seem to be antagonistic to inconclusive tests (https://xunit.codeplex.com/workitem/9691).
I think you can use Assume.That in your munitions test method it won't work while setup but you will to use that in test method.
Is there a way to run a single test in isolation, or a group of tests?
I find that when I am trying to debug a particular test, it's really annoying because previous test cases may trigger that same break point-- So I typically comment out all my tests except the one I am working on, but that seems so ghetto... So I am wondering if there's a better way?
The only way I know of doing this is by editing the scheme. The "Test" action has a bunch of checkboxes where you can turn on or off groups of tests (i.e. classes) or single tests. If you're doing this with the same tests a lot, you could create a set of schemes for different groups and set it up just the way you want. (Or just edit it each time.)
Perhaps not the answer you're looking for, but in AppCode, the iOS and OSX IDE from Jetbrains you put the cursor anywhere over the test you want, and then trigger it with a keyboard shortcut. (I use IntelliJ mapping so its Ctrl+Shift+F10).
It also has a great test runner showing green/red bars for tests that pass or fail.
I am writing functional tests using TestNG, and I have a few dozens of similar tests with different data. I would like to use DataProvider to reduce repeating code.
But some of those tests pass, some fail (due to a known defect). I want to disable failing tests until they are fixed, so they don't spoil whole picture of test run.
I see that AnnotationTransformer can change test annotations dynamically. Can AnnotationTransformer disable test only with some of the data sets? Or will it disable test with all provided data and it is better not to change anything?
Thanks in advance.
Why not simply put these failing tests in a group, say "broken", and exclude that test from your runs? Much simpler than using an annotation transformer, and the reports will show you which groups were excluded, so there is no risk to miss any when comes the time to ship.
We use TestNG and Selenium WebDriver to test our web application.
Now our problem is that we often have several tests that need to run in a certain order, e.g.:
login to application
enter some data
edit the data
check that it's displayed correctly
Now obviously these tests need to run in that precise order.
At the same time, we have many other tests which are totally independent from the list of tests above.
So we'd like to be able to somehow put tests into "groups" (not necessarily groups in the TestNG sense), and then run them such that:
tests inside one "group" always run together and in the same order
but different test "groups" as a whole can run in any order
The second point is important, because we want to avoid dependencies between tests in different groups (so different test "groups" can be used and developed independently).
Is there a way to achieve this using TestNG?
Solutions we tried
At first we just put tests that belong together into one class, and used dependsOnMethods to make them run in the right order. This used to work in TestNG V5, but in V6 TestNG will sometimes interleave tests from different classes (while respecting the ordering imposed by dependsOnMethods). There does not seem to be a way to tell TestNG "Always run tests from one class together".
We considered writing a method interceptor. However, this has the disadvantage that running tests from inside an IDE becomes more difficult (because directly invoking a test on a class would not use the interceptor). Also, tests using dependsOnMethods cannot be ordered by the interceptor, so we'd have to stop using that. We'd probably have to create our own annotation to specify ordering, and we'd like to use standard TestNG features as far as possible.
The TestNG docs propose using preserve-order to order tests. That looks promising, but only works if you list every test method separately, which seems redundant and hard to maintain.
Is there a better way to achieve this?
I am also open for any other suggestions on how to handle tests that build on each other, without having to impose a total order on all tests.
PS
alanning's answer points out that we could simply keep all tests independent by doing the necessary setup inside each test. That is in principle a good idea (and some tests do this), however sometimes we need to test a complete workflow, with each step depending on all previous steps (as in my example). To do that with "independent" tests would mean running the same multi-step setup over and over, and that would make our already slow tests even slower. Instead of three tests doing:
Test 1: login to application
Test 2: enter some data
Test 3: edit the data
we would get
Test 1: login to application
Test 2: login to application, enter some data
Test 3: login to application, enter some data, edit the data
etc.
In addition to needlessly increasing testing time, this also feels unnatural - it should be possible to model a workflow as a series of tests.
If there's no other way, this is probably how we'll do it, but we are looking for a better solution, without repeating the same setup calls.
You are mixing "functionality" and "test". Separating them will solve your problem.
For example, create a helper class/method that executes the steps to log in, then call that class/method in your Login test and all other tests that require the user to be logged in.
Your other tests do not actually need to rely on your Login "Test", just the login class/method.
If later back-end modifications introduce a bug in the login process, all of the tests which rely on the Login helper class/method will still fail as expected.
Update:
Turns out this already has a name, the Page Object pattern. Here is a page with Java examples of using this pattern:
http://code.google.com/p/selenium/wiki/PageObjects
Try with depends on group along with depends on method. Add all methods in same class in one group.
For example
#Test(groups={"cls1","other"})
public void cls1test1(){
}
#Test(groups={"cls1","other"}, dependsOnMethods="cls1test1", alwaysrun=true)
public void cls1test2(){
}
In class 2
#Test(groups={"cls2","other"}, dependsOnGroups="cls1", alwaysrun=true)
public void cls2test1(){
}
#Test(groups={"cls2","other"}, dependsOnMethods="cls2test1", dependsOnGroups="cls1", alwaysrun=true)
public void cls2test2(){
}
There is an easy (whilst hacky) workaround for this if you are comfortable with your first approach:
At first we just put tests that belong together into one class, and used dependsOnMethods to make them run in the right order. This used to work in TestNG V5, but in V6 TestNG will sometimes interleave tests from different classes (while respecting the ordering imposed by dependsOnMethods). There does not seem to be a way to tell TestNG "Always run tests from one class together".
We had a similar problem: we need our tests to be run class-wise because we couldn't guarantee the test classes not interfering with each other.
This is what we did:
Put a
#Test( dependsOnGroups= { "dummyGroupToMakeTestNGTreatThisAsDependentClass" } )
Annotation on an Abstract Test Class or Interface that all your Tests inherit from.
This will put all your methods in the "first group" (group as described in this paragraph, not TestNG-groups). Inside the groups the ordering is class-wise.
Thanks to Cedric Beust, he provided a very quick answer for this.
Edit:
The group dummyGroupToMakeTestNGTreatThisAsDependentClass actually has to exist, but you can just add a dummy test case for that purpose..
Using OCUnit & Xcode, is there a way of running just one test?
Ideally, I'd be able to run just one test method, but if there's a way to just run a single test case, that would be OK too.
What I'm currently doing is running the 'Test' task which runs all of my tests, but this takes up a lot of time, which ideally could be spent doing other things.
See this post from an Xcode engineer:
http://chanson.livejournal.com/119578.html
The last paragraph explains how to specify a single test case class.
For more info see Chris' entire series on unit testing:
http://chanson.livejournal.com/tag/unit%20testing