We have several cucumber step definitions that are modifying the database which would mess up the test afterwards if it doesn't get cleaned up after the test runs. We do this by having a function with the #After annotation that will clean things up.
The problem is that if there's a failure in one of the tests, the function with #After doesn't run, which leaves the database in a bad state.
So the question is, how can I make sure the function with #After always runs, regardless if a test failed or not?
I saw this question, but it's not exactly what I'm trying to do, and the answers don't help.
If it helps, here is part of one of the tests. It's been greatly stripped down, but it has what I think are the important parts.
import static org.hamcrest.MatcherAssert.assertThat;
import cucumber.api.java.After;
public class RunMacroGMUStepDefinition
{
#Autowired
protected ClientSOAPRecordkeeperInterface keeper;
#Given( "^the following Macro exists:$" )
#Transactional
public void establishDefaultPatron( final DataTable dataTable )
{
for ( final DataTableRow dataTableRow : dataTable.getGherkinRows() )
{
// Stuff happens here
keeper.insert( macroScriptRecord );
}
}
#After( value = "#RunMacroGMU" )
#Transactional
public void teardown()
{
for ( int i = 0; i < macroScripts.size(); i++ )
{
keeper.delete( macroScripts.get( i ) );
}
}
// Part of #Then
private void compareRecords( final String has, // Other stuff )
{
// Stuff happens here
if ( has.equals( "include" ) )
{
assertThat( "No matching data found", foundMatch, equalTo( true ) );
}
else
{
assertThat( "Found matching data", foundMatch, equalTo( false ) );
}
}
}
I personally use Behat (The PHP dist of Cucumber), and we use something like this to take screenshots after a failed test. Did a bit of searching, and found this snippet in Java, that may help with this situation.
#After
public void tearDown(Scenario scenario) {
if (scenario.isFailed()) {
(INSERT FUNCTIONS YOU WOULD LIKE TO RUN AFTER A FAILING TEST HERE)
}
driver.close();
}
I hope this helps.
Related
public enum SystemConstants
{
SystemTypeDocument,
ApplicationTypeDocument
}
public interface ISystemBaseObject
{
SystemConstants SystemType();
}
public class ExploreMockExample
{
ISystemBaseObject systemBaseObject;
public ExploreMockExample(ISystemBaseObject systemObject)
{
systemBaseObject = systemObject;
}
public int MethodToBeTested()
{
if (systemBaseObject.SystemType() == SystemConstants.SystemTypeDocument)
{
return 1;
}
else
{
return 2;
}
}
}
Using intellitest along with NUnit3.
When I right click MethodToBeTested, and then select run intellitest, expected outcome is Intellitest test should achieve maximum code coverage and create test case with valid test data to cover both if (systemBaseObject.SystemType() == SystemConstants.SystemTypeDocument) and else branch statement.
Some blogs suggested to create factory for class and create mock object of interface. And use PexChoose static method to allow pex framework to explore code to achieve maximum code coverage.
[PexFactoryMethod(typeof(ExploreMockExample))]
public static ExploreMockExample CreateMock()
{
var mockComosBaseObject = new Mock<ISystemBaseObject>();
mockComosBaseObject.Setup(c =>c.SystemType()).
Returns(PexChoose.EnumValue<SystemConstants>(nameof(SystemConstants)));
return new ExploreMockExample(mockComosBaseObject.Object);
}
With above setup, Intellitest could above to generate only one test case which is covering if statement, if (systemBaseObject.SystemType() == SystemConstants.SystemTypeDocument).
What can be done, to allow intellitest to create test case which will cover else statement having result value as 2.
Create mock implementation for your interface. Like mentioned below,
public class SystemBaseObject : ISystemBaseObject
{
public SystemConstants SystemType()
{
return PexChoose.EnumValue<SystemConstants>("SystemConstants");
}
}
PexChoose will help intellitest to explore code, return value will depend upon uses of SystemConstants in original class.
Then, create factory of ExploreMockExample using SystemBaseObject,
[PexFactoryMethod(typeof(ExploreMockExample))]
public static ExploreMockExample Create()
{
return new ExploreMockExample(new SystemBaseObject());
}
Run Intellitest on actual method MethodToBeTested, Intellitest will create 2 unit test case to cover both if else branch statement.
First test case,
[PexGeneratedBy(typeof(ExploreMockExampleTest))]
public void MethodToBeTested806()
{
ExploreMockExample exploreMockExample;
int i;
exploreMockExample = ExploreMockExampleFactory.Create();
i = this.MethodToBeTested(exploreMockExample);
PexAssert.AreEqual<int>(1, i);
PexAssert.IsNotNull((object)exploreMockExample);
}
Kindly observe PexAssert.AreEqual(1, i), which will cover if branch.
Second test case,
[PexGeneratedBy(typeof(ExploreMockExampleTest))]
public void MethodToBeTested792()
{
ExploreMockExample exploreMockExample;
int i;
exploreMockExample = ExploreMockExampleFactory.Create();
IPexChoiceRecorder choices = PexChoose.Replay.Setup();
choices.NextSegment(1).DefaultSession
.At(0, "SystemConstants", (object)(SystemConstants.ApplicationTypeDocument));
i = this.MethodToBeTested(exploreMockExample);
PexAssert.AreEqual<int>(2, i);
PexAssert.IsNotNull((object)exploreMockExample);
}
Kindly observe PexAssert.AreEqual(2, i), which will cover else branch.
Using PexChoose.Replay.Setup() it will return IPexChoiceRecorder, and it will make choice about to have SystemConstants.ApplicationTypeDocument as argument value to cover else block.
I'm trying to test the following class (I've left out the implementation)
public class UTRI implements UTR {
public void runAsUser(String userId, Runnable r);
}
This is the way I would use it:
UTRI.runAsUser("User1", new Runnable () {
private void run() {
//do whatever needs to be done here.
}
});
The problem is, I don't know how to use EasyMock to test functions that return void. That and I'm also not too familiar with testing in general (right out of school!). Can someone help explain to me what I need to do to approach this? I was thinking about making the UTRI a mock and doing expectlastcall after that, but realistically, not sure.
public class UTRITest {
UTRI utri = new UTRI();
#Test
public void testRunAsUser() {
// Create Mocks
Runnable mockRunnable = EasyMock.createMock(Runnable.class);
// Set Expectations
**mockRunnable.run();
EasyMock.expectLastCall().once();**
EasyMock.replay(mockRunnable);
// Call the method under test
utri.runAsUser("RAMBO", **mockRunnable**);
// Verify if run was called on Runnable!!
EasyMock.verify(mockRunnable);
}
}
How to test that the correct exception is thrown when using Cucumber JVM? When using JUnit, I would do something like this:
#Test(expected = NullPointerException.class)
public void testExceptionThrown(){
taskCreater.createTask(null);
}
As you can see, this is very elegant. But how can I achieve the same elegance, when using cucumber JVM? My test looks like this right now:
#Then("the user gets a Null pointer exception$")
public void null_exception_thrown() {
boolean result = false;
try {
taskCreater.createTask(null);
} catch (NullPointerException e) {
result = true;
}
assertTrue(result);
}
Note the need for a try..catch followed by an assertTrue on a flag.
Testing the not-happy-path can be hard. Here's a nice way that I've found to do it with cucumber.
Scenario: Doing something illegal should land you in jail
Then a failure is expected
When you attempt something illegal
And it fails.
OK, don't shoot me because I put the Then before the When, I just think it reads better but you don't have to do that.
I store my excepions in a (cucumber-scoped) world object, but you could also do it in your step file, but this will limit you later.
public class MyWorld {
private boolean expectException;
private List<RuntimeException> exceptions = new ArrayList<>();
public void expectException() {
expectException = true;
}
public void add(RuntimeException e) {
if (!expectException) {
throw e;
}
exceptions.add(e);
}
public List<RuntimeException> getExceptions() {
return exceptions;
}
}
Your steps are then pretty simple:
#Then("a failure is expected")
public void a_failure_is_expected() {
myWorld.expectException();
}
In a step where you are (at least sometimes) expecting an exception, catch it and add it to the world.
#When("you attempt something illegal")
public void you_attempt_something_illegal() {
try {
myService.doSomethingBad();
} catch (RuntimeException e) {
world.add(e);
}
}
Now you can check whether the exception was recorded in the world.
#And("it fails")
public void it_fails() {
assertThat(world.getExceptions(), is(not(empty()));
}
The most valuable thing about this approach is that it won't swallow an exception when you don't expect it.
Have you tried using the junit #Rule annotation with ExpectedException, like this:
#Rule
public ExpectedException expectedEx = ExpectedException.none();
#Then("the user gets a Null pointer exception$")
public void null_exception_thrown() {
expectedEx.expect(NullPointerException.class);
//expectedEx.expectMessage("the message");
taskCreater.createTask(null);
}
Behaviour testing verifies if your project follows specifications, and I doubt the user expects a NullPointerException while using the system.
In my opinion (I'm not familiar with your project), exceptions should only be checked during Unit Tests, as they correspond to an unexpected error or user mistake.
It's very unusual to check for exceptions during behaviour tests. If an exception is thrown during a test, it should fail.
for instance:
test.feature
Given that I have a file "users.txt"
And I try to import users /* If an Exception is thrown here, the test fails */
Then I should see the following users: /* list of users */
In my unit test, I would have:
#Test(expected = MyException.class)
public void importUsersShouldThrowMyExceptionWhenTheFileIsNotFound() {
// mock a call to a file and throw an exception
Mockito.when(reader.readFile("file.txt").thenThrow(new FileNotFoundException());
importer.importUsers();
}
I suggest you to use org.assertj.assertj-core.
Thanks to Assertions class, you could simplify your assertion like bellow:
#Then("the user gets a Null pointer exception$")
public void null_exception_thrown() {
Assertions.assertThatThrownBy(() -> taskCreater.createTask(null)).
isInstanceOf(NullPointerException.class);
}
Never used Cucumber, but would
public void null_exception_thrown() {
try {
taskCreater.createTask(null);
fail("Null Pointer Expected");
} catch (NullPointerException e) {
// Do Nothing
}
}
work for you?
I believe Cucumber is meant to test higher level acceptance tests than lower level unit tests as in that case we will be testing the structure opposed to the behavior which is not the desired usage of the Cucumber framework.
i used to have expected exception name in feature steps. for a calculator test example
Here is my feature file entry for division exception :
Scenario: Dividing a number with ZERO
Given I want to test calculator
When I insert 5.5 and 0.0 for division
Then I got Exception ArithmeticException
So, you may see, I have added the exception name. So, in step definition , i get the name of the exception.
Now, we need to get the exception when dividing by ZERO and put in a variable.
And get this variable to compare its class name(exception name)
so, in step definition where i divide by zero
private Exception actualException;
#When("^I insert (.+) and (.+)for division$")
public void iInsertAndForDivision(double arg0, double arg1) throws Throwable {
try {
result = calculator.div(arg0, arg1);
} catch (Exception e) {
actualException =e;
}
}
And i need the step definition for validating the exception
#Then("^I got Exception (.+)")
public void iGotArithmeticException(String aException) throws Throwable {
Assert.assertEquals("Exception MissMatch",aException,actualException.getClass().getSimpleName());
}
The complete project can be seen here : Steps for cucumber
I need to write the testcase Pass/Fail status in an Excel report. How do I catch the result in a simple way? In Ant XML/HTML reports status (Success/Failed) is displayed so I believe there must be some way to catch this..may be in the #After Class.
Can anyone help me out with this?
I am using Selenium Webdriver with JUnit4 and using Apache POI(oh its screwing my mind too!) for excel handling. Let me know in case you need more info.
Thanks :)
P.S: As I am asking a lot of questions it would be great if someone can change or suggest me changes to make these questions and threads helpful for others.
I got your problem and this is what I have do in all my test cases now -
In your test case if you want to check if 2 Strings are equal -
You might be using this code -
Assert.assertTrue(value1.equals(value2));
And if these values are not equal an AssertionError is generated.
Instead you can change your code like this -
String testCaseStatus = "";
if(value1.equals(value2)){
testCaseStatus = "success";
}
else{
testCaseStatus = "fail";
}
Now you store this result in your excel sheet by passing the testCaseStatus to your code which adds a line to your excel which you have implemented using Apache POI. You can also handle the conditions of "error" if you implement a try catch block.
For example you can catch some exception and add the status as error to your excel sheet.
Edited part of answer -
I just figured out on how to use the TestResult class -
This is some sample code I'm posting -
This is the test case class called ExampleTest -
public class ExampleTest implements junit.framework.Test {
#Before
public void setUp() throws Exception {
}
#After
public void tearDown() throws Exception {
}
#Test
public void test(TestResult result) {
try{
Assert.assertEquals("hari", "");
}catch(AssertionFailedError e){
result.addFailure(this, e);
}catch(AssertionError e){
result.addError(this, e);
}
}
#Override
public int countTestCases() {
// TODO Auto-generated method stub
return 0;
}
#Override
public void run(TestResult result) {
test(result);
}
}
I call the above test case from this code -
public class Test {
public static void main(String[] args) {
TestResult result = new TestResult();
TestSuite suite = new TestSuite();
suite.addTest(new ExampleTest());
suite.run(result);
System.out.println(result.errorCount());
}
}
You can call many test cases by just adding them to this suite and then get the entire result using the TestResult class failures() and errors() methods.
You can read more on this from here.
We have a set of integration test which depend upon same set of static data. Since the amount of data is huge we dont want to set it up per test level. Is it possible to setup data at the start, run group of test and rollback the data at the end of test.
What we effectively want is the rollback at test suite level rather than test case level. We are using grails 1.3.1, any pointers would be highly helpful for us to proceed. Thanks in advance.
-Prakash
for one test case you could use:
#BeforeClass
public static void setUpBeforeClass() throws Exception {
}
#AfterClass
public static void tearDownAfterClass() throws Exception {
}
haven't tried a suite of test cases (yet).
i did have some trouble using findByName in the static methods and had to resort to saving an id and using get.
i did try rolling up a suite, but no joy, getting a: no runnable methods.
You can take control of the transaction/rollback behaviour by marking your test case as non-transactional and managing data, transactions and rollbacks yourself. Example:
class SomeTests extends GrailsUnitTestCase {
static transactional = false
static boolean testDataGenerated = false
protected void setUp() {
if (!testDataGenerated) {
generateTestData()
testDataGenerated = true
}
}
void testSomething() {
...test...
}
void testSomethingTransactionally() {
DomainObject.withTransaction {
...test...
}
}
void testSomethingTransactionallyWithRollback() {
DomainObject.withTransaction { status ->
...test...
status.setRollbackOnly()
}
}
}