I have a project that builds with maven2 and runs a series of JUnit test cases against the code. This has worked fine up until this point, where I now have 2 tests that must run in a certain sequence for things to work correctly, let's say TestA and Test (A then B). Unfortunately, maven2 doesn't understand this, so I'm looking for a way of convincing it that it needs to run the tests in this order.
The problem is that I'm setting some final static fields in TestB, but I'm doing this from TestA, which itself uses those fields, and successful execution of the test depends on those fields being set to their new values (there is absolutely no way around this, otherwise I would have taken that road long before now). So, it is imperative that TestA loads first, and it will of course cause TestB to be loaded when tries to access it. However, maven2 has decided that it will run TestB then TestA, which means those final fields are already set and cannot be changed.
So what I'm looking for is either a way to specify the order in which the tests are executed (A then B, every time), or a way to easily cause TestB to be reloaded by whichever classloader that JUnit uses.
EDIT - one other option might be some option like the old JUnit GUI tool has, that causes all classes to be reloaded for each test. I have looked and looked and not found such a flag in the maven junit plugin, if such a thing exists, then that would also work.
Fork mode can force each test to run in its own JVM so every class is re-loaded for each test.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<forkMode>pertest</forkMode>
</configuration>
</plugin>
The test order in JUnit is intentionally undefined, this is not a Maven issue and it's probably just luck that your tests were running OK up until now.
Sal's answer addresses your question directly, however forking the JVM on each test for large test counts can hugely increase your build time.
An alternative approach would be to use a testing library such as PowerMock (it currently works with EasyMock and Mockito) to clear the static fields in TestB's initialisation, this avoids the need for any JVM forking, and ensures your tests are portable.
From the PowerMock website:
PowerMock is a framework that extend other mock libraries such as EasyMock with more powerful capabilities. PowerMock uses a custom classloader and bytecode manipulation to enable mocking of static methods, constructors, final classes and methods, private methods, removal of static initializers and more. By using a custom classloader no changes need to be done to the IDE or continuous integration servers which simplifies adoption. Developers familiar with EasyMock will find PowerMock easy to use, since the entire expectation API is the same, both for static methods and constructors. PowerMock extends the EasyMock API with a small number of methods and annotations to enable the extra features. From version 1.1 PowerMock also has basic support for Mockito.
Related
I was thinking of implementing automated tests for different part of an ActivePivot Servers and most importantly post-processors.
Since I am at the very beginning, I would like to know more about the state of the art in this field, what are the best practices and if there are any caveats to avoid.
IF you have any experience, I will be delighted to read from you.
Cheers,
Pascal
That is a very broad question. An ActivePivot solution is a piece of java software and inherits from all the best practices regarding the testing and continuous build of a software project.
But here are some basic ActivePivot entry points:
How, where and when to write tests?
Write junit tests, run them with maven, setup continuous build with Jenkins.
How to embbed a (real, non trivial) ActivePivot instance within a unit test?
Start an embbeded Jetty web application server. The ActivePivot Sandbox application is an example of that (look at com.quartetfs.pivot.jettyserver.JettyServer). If you want a series of unit tests to run against the same ActivePivot instance, you can start the Jetty server statically (for instance in a static method annotated with #BeforeClass). In any case don't forget to stop it at the end of the tests.
How to write performance tests?
In the Sandbox project, there is a small MDX benchmark called com.quartetfs.pivot.client.MDXBenchmark. It is easy to enrich and a good starting point. There is also com.quartetfs.pivot.client.WebServiceClient that illustrates connecting to ActivePivot
How to test post processors?
As of ActivePivot release 4.3.5 there is no framework dedicated to isolated post processor testing. Post processors are tested through queries (MDX queries or GetAggregates queries). Of course if your post processor implementation has some utility methods, those can be tested one by one in standard unit tests.
To test an ActivePivot-based project, the simpler is to re-use your Spring configuration. This can be done with ClassPathXmlApplicationContext:
ApplicationContext context = new ClassPathXmlApplicationContext("beans.xml");
This simple test will check if your Spring is actually Ok. Then, if you want to run a query, you could do the following:
IQueriesService queriesService = context.getBean(IQueriesService.class);
queriesService.execute(new MDXQuery(someMDX));
If you want to check your loading layer, you can do:
IStoreUniverse storeUniverse = context.getBean(IStoreUniverse.class);
for (IRelationalStore store : storeUniverse.values) {
assertEquals(hardcodedValue1, store.getSize())
assertEquals(hardcodedValue2, store.search("someKey", "someValue").size())
}
This way, you don't need to start a web-app container, which may fail because it needs some port to be available (meaning for instance you can't run several tests at the same time).
Post-Processors should be either Basic or DynamicAggregation post-processors, which are easy to test: focus on .init and the evaluation methods called on point ILocations. Advanced Post-processors can not be reasonnably unit-tested. Then, I advice writing MDX queries as simple as possible but relevant given the Post-Processor.
Any unit-test framework and mock framework could be used. Still, I advice using JUnit and Mockito.
I would recommend using Spring4JUnit to launch the context. You can then autowire the beans and access things like the queries service and the active pivot manager directly.
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(locations = {"classpath:SPRING-INF/ActivePivot.xml", "classpath:cusomTestContext-test.xml"})
...
#Resource
private IActivePivotManager manager;
#Resource
private IQueriesService queriesService;
#Test
public void testManagerOk() {
assertNotNull(manager);
assertTrue(manager.getStatus().equals(State.STARTED));
}
#Test
public void testManagerOk() {
// run a query with the queries service
}
...
You can define custom test properties for the tests in a separate context file, say for loading a set of test data.
I need to run tests in order. I fail to find this adequately documented anywhere. I would prefer to do this from command line. Something like
mvn -Dtest=test1,test2,test3,test5 test
How do I do this?
You can't specify the run order of your tests.
A workaround to do this is to set the runOrder parameter to alphabetical.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<runOrder>alphabetical</runOrder>
</configuration>
</plugin>
and then you need to have rename your tests to obtain the expected order.
However it isn't a good idea to have dependent tests. Unit tests must be fIrst.
You could create a test suite which runs all of your tests and run that.
With junit 4: -
#RunWith(Suite.class)
#Suite.SuiteClasses({Test1.class,
Test2.class,
Test3.class,
Test4.class,
Test5.class
})
public class TestSuite
{
}
That will run them in the correct order.
If you really need an order of your tests than you should use testng instead of JUnit where you can define dependencies between tests and based on that a particular order of tests. I know that in practice their are times where the independent paradigm does not work.
There is a Maven Surefire plugin that allows you to specify test order.
On the off chance that your tests need to be run in order because they depend on each other, I would strongly recommend against that. Each test should be independent and able to be run by itself. And if each test is independent then it doesn't matter what order they run in. Having independent tests also means you can run a single test repeatedly without having to rerun the entire test chain. This is a huge time savings.
The runOrder parameter of the surefire-plugin will help, but it will only help if the classes are executed in order. It does not help to order the test methods within one class (surefire-plugin 2.22.2, junit 5.6.1). To achieve the order within a test class with jUnit use jUnit's feature of controlling the test order within a class by #TestMethodOrder(MethodOrderer.Alphanumeric.class) (jUnit 5).
If your unit tests need to be ran in a specific order, it's probably because your tests are badly designed, or your app is badly designed. Your unit tests should be independant from each other.
We use TestNG and Selenium WebDriver to test our web application.
Now our problem is that we often have several tests that need to run in a certain order, e.g.:
login to application
enter some data
edit the data
check that it's displayed correctly
Now obviously these tests need to run in that precise order.
At the same time, we have many other tests which are totally independent from the list of tests above.
So we'd like to be able to somehow put tests into "groups" (not necessarily groups in the TestNG sense), and then run them such that:
tests inside one "group" always run together and in the same order
but different test "groups" as a whole can run in any order
The second point is important, because we want to avoid dependencies between tests in different groups (so different test "groups" can be used and developed independently).
Is there a way to achieve this using TestNG?
Solutions we tried
At first we just put tests that belong together into one class, and used dependsOnMethods to make them run in the right order. This used to work in TestNG V5, but in V6 TestNG will sometimes interleave tests from different classes (while respecting the ordering imposed by dependsOnMethods). There does not seem to be a way to tell TestNG "Always run tests from one class together".
We considered writing a method interceptor. However, this has the disadvantage that running tests from inside an IDE becomes more difficult (because directly invoking a test on a class would not use the interceptor). Also, tests using dependsOnMethods cannot be ordered by the interceptor, so we'd have to stop using that. We'd probably have to create our own annotation to specify ordering, and we'd like to use standard TestNG features as far as possible.
The TestNG docs propose using preserve-order to order tests. That looks promising, but only works if you list every test method separately, which seems redundant and hard to maintain.
Is there a better way to achieve this?
I am also open for any other suggestions on how to handle tests that build on each other, without having to impose a total order on all tests.
PS
alanning's answer points out that we could simply keep all tests independent by doing the necessary setup inside each test. That is in principle a good idea (and some tests do this), however sometimes we need to test a complete workflow, with each step depending on all previous steps (as in my example). To do that with "independent" tests would mean running the same multi-step setup over and over, and that would make our already slow tests even slower. Instead of three tests doing:
Test 1: login to application
Test 2: enter some data
Test 3: edit the data
we would get
Test 1: login to application
Test 2: login to application, enter some data
Test 3: login to application, enter some data, edit the data
etc.
In addition to needlessly increasing testing time, this also feels unnatural - it should be possible to model a workflow as a series of tests.
If there's no other way, this is probably how we'll do it, but we are looking for a better solution, without repeating the same setup calls.
You are mixing "functionality" and "test". Separating them will solve your problem.
For example, create a helper class/method that executes the steps to log in, then call that class/method in your Login test and all other tests that require the user to be logged in.
Your other tests do not actually need to rely on your Login "Test", just the login class/method.
If later back-end modifications introduce a bug in the login process, all of the tests which rely on the Login helper class/method will still fail as expected.
Update:
Turns out this already has a name, the Page Object pattern. Here is a page with Java examples of using this pattern:
http://code.google.com/p/selenium/wiki/PageObjects
Try with depends on group along with depends on method. Add all methods in same class in one group.
For example
#Test(groups={"cls1","other"})
public void cls1test1(){
}
#Test(groups={"cls1","other"}, dependsOnMethods="cls1test1", alwaysrun=true)
public void cls1test2(){
}
In class 2
#Test(groups={"cls2","other"}, dependsOnGroups="cls1", alwaysrun=true)
public void cls2test1(){
}
#Test(groups={"cls2","other"}, dependsOnMethods="cls2test1", dependsOnGroups="cls1", alwaysrun=true)
public void cls2test2(){
}
There is an easy (whilst hacky) workaround for this if you are comfortable with your first approach:
At first we just put tests that belong together into one class, and used dependsOnMethods to make them run in the right order. This used to work in TestNG V5, but in V6 TestNG will sometimes interleave tests from different classes (while respecting the ordering imposed by dependsOnMethods). There does not seem to be a way to tell TestNG "Always run tests from one class together".
We had a similar problem: we need our tests to be run class-wise because we couldn't guarantee the test classes not interfering with each other.
This is what we did:
Put a
#Test( dependsOnGroups= { "dummyGroupToMakeTestNGTreatThisAsDependentClass" } )
Annotation on an Abstract Test Class or Interface that all your Tests inherit from.
This will put all your methods in the "first group" (group as described in this paragraph, not TestNG-groups). Inside the groups the ordering is class-wise.
Thanks to Cedric Beust, he provided a very quick answer for this.
Edit:
The group dummyGroupToMakeTestNGTreatThisAsDependentClass actually has to exist, but you can just add a dummy test case for that purpose..
I have a Maven/Java project I've been working on for years, and I wanted to take JavaPosse's advice and start writing my tests in Scala. I've written a few tests following ScalaTest's JUnit4 quick start, and now I want these tests to be executed while running "mvn test". How should I do this? What should I put into my pom.xml to allow the tests in src/test/scala to be run side-by-side my old JUnit4 tests?
Cheers
Nik
PS, yes, I've been Googling, but all I could find on the topic were some pre-v1.0 suggestions that I didn't get working
PPS, bonus question: how can I run these tests one-at-a-time by rightclicking them in Eclipse/STS and say "Debug As... ScalaTest" or something similar where I've so far said "Debug As... JUnit Test"?
PPPS, I expect the answer has changed since July '09?
The second answer in one of the questions you linked to SHOULD work:
Is there a Scala unit test tool that integrates well with Maven?
You annotate your tests with a junit #RunWith annotation and give it the scalatest http://www.artima.com/docs-scalatest-2.0.RC3/#org.scalatest.junit.JUnitRunner
If your Tests also adhere to any naming conventions possibly enforced by Maven, this should work fine.
Note: It doesn't matter what kind of scalatest trait you use. All of them should work. If they don't and Bill Venners doesn't answer to this question, contact him on the ScalaTest mailing list.
Other Note: you can run such test suites in Eclipse using the normal JUnit plugin. But you can't run single tests, since the plugin expects to deduct a method name from the test name, which doesn't work with all types of scalatest tests.
I'm wondering if there is any solution to let Scala tests run automatically upon change of test class itself or class under the test (just to test automatically pairs Class <---> ClassTest) would be a good start.
sbt can help you with this. After you setup project, just run
~test
~ means continuous execution. So that sbt will watch file system changes and when changes are detected it recompiles changed classes and tests your code. ~testQuick can be even more suitable for you, because it runs only tests, that were changed (including test class and all it's transitive dependencies). You can read more about this here:
http://code.google.com/p/simple-build-tool/wiki/TriggeredExecution
http://php.jglobal.com/blog/?p=363
By the way, ~ also works with other tasks like ~run.