Execute some action when Spock test fails - testing

I'd like to execute some action when Spock test fails. Specifically, take a screenshot. Is it possible? How to do it?

Create a listener class
class ExampleListener extends AbstractRunListener {
def void error(ErrorInfo error) {
println "Actual on error logic"
}
}
then add it to each specification using implementation of IGlobalExtension that is executed for each Spec
class GlobalSpecExtension implements IGlobalExtension {
#Override
void visitSpec(SpecInfo specInfo) {
specInfo.addListener(new ExampleListener())
}
}
and finally create file named org.spockframework.runtime.extension.IGlobalExtension in your META-INF/services directory (typically it will be under src/test/resources if you are using Maven) with the full name of your IGlobalExtension implementation e.g.
com.example.tests.GlobalSpecExtension

The best way to achieve this is to write a (global or annotation-driven) Spock extension that implements and registers an AbstractRunListener. For an example, see OptimizeRunOrderExtension. For how to register a global extension, see the IGlobalExtension descriptor.
There isn't much documentation on extensions because the APIs are still subject to change. If you want to play it safe (and can live with some restrictions), you can implement a JUnit Rule instead.
One problem that you may encounter in both cases is that they don't provide access to the current spec instance. If you need this, you may have to use both an AbstractRunListener (to be notified of the failure) and an IMethodInterceptor (to get hold of the spec instance), both registered by the same extension. (Shouldn't be this hard, but that's what's currently there.)

I've managed to do it this way:
class ExampleTest extends GebSpec{
static boolean success = false
def setup(){
success = false
}
def cleanup(){
assert success == true, someAction()
}
def someAction(){
}
def "TestCase"(){
expect:
/*What you expect here*/
(success = true) != null
}
}
Before each test case "success" is set to false by the setup() method.
At the end of each test case you add the "(success = true) != null" statement. Therefore "success" will only be true if the test case has passed.
After each test case the cleanup() method will verify if "success" is true. If it isn't the method someAction() will be called.

I can't upvote or comment on user3074543's answer, but it's simpler than creating an extension. I want easy. So I shortened user*'s a little (I don't mean the 1-line methods). You can make the logic simpler by recording failure instead of success, and reduce typing with a done() helper.
class Test extends spock.lang.Specification {
def fail
def setup(){ fail = true }
def done(){ !(fail = false) }
def cleanup(){ fail && doStuffWhenFail() }
def 'test things'(){
expect:
stuff
done()
}
}

Related

How to Take Screenshot when TestNG Assert fails?

String Actualvalue= d.findElement(By.xpath("//[#id=\"wrapper\"]/main/div[2]/div/div[1]/div/div[1]/div[2]/div/table/tbody/tr[1]/td[1]/a")).getText();
Assert.assertEquals(Actualvalue, "jumlga");
captureScreen(d, "Fail");
The assert should not be put before your capture screen. Because it will immediately shutdown the test process so your code
captureScreen(d, "Fail");
will be not reachable
This is how i usually do:
boolean result = false;
try {
// do stuff here
result = true;
} catch(Exception_class_Name ex) {
// code to handle error and capture screen shot
captureScreen(d, "Fail");
}
# then using assert
Assert.assertEquals(result, true);
1.
A good solution will be is to use a report framework like allure-reports.
Read here:allure-reports
2.
We don't our tests to be ugly by adding try catch in every test so we will use Listeners which are using an annotations system to "Listen" to our tests and act accordingly.
Example:
public class listeners extends commonOps implements ITestListener {
public void onTestFailure(ITestResult iTestResult) {
System.out.println("------------------ Starting Test: " + iTestResult.getName() + " Failed ------------------");
if (platform.equalsIgnoreCase("web"))
saveScreenshot();
}
}
Please note I only used the relevant method to your question and I suggest you read here:
TestNG Listeners
Now we will want to take a screenshot built in method by allure-reports every time a test fails so will add this method inside our listeners class
Example:
#Attachment(value = "Page Screen-Shot", type = "image/png")
public byte[] saveScreenshot(){
return ((TakesScreenshot)driver).getScreenshotAs(OutputType.BYTES);
}
Test example
#Listeners(listeners.class)
public class myTest extends commonOps {
#Test(description = "Test01: Add numbers and verify")
#Description("Test Description: Using Allure reports annotations")
public void test01_myFirstTest(){
Assert.assertEquals(result, true)
}
}
Note we're using at the beginning of the class an annotation of #Listeners(listeners.class) which allows our listeners to listen to our test, please mind the (listeners.class) can be any class you named your listeners.
The #Description is related to allure-reports and as the code snip suggests you can add additional info about the test.
Finally, our Assert.assertEquals(result, true) will take a screen shot in case the assertion fails because we enabled our listener.class to it.

Run specific Spock Tests without setup?

In the documentation Spock mentions these function in the section 'Fixture Methods':
def setup() {} // run before every feature method
def cleanup() {} // run after every feature method
def setupSpec() {} // run before the first feature method
def cleanupSpec() {} // run after the last feature method
I am using the setup function to initialize new objects for all tests. However some tests are supposed to test what happens when there are no other objects yet.
Because of the setup these tests fail now.
Is there any way to suspend the setup for certain functions? An annotation perhaps?
Or do I have to create a separate "setup" function and call them in each test I do. The majority of tests uses it!
You can always override value inside your test method only for that specific method. Take a look at following example:
import spock.lang.Shared
import spock.lang.Specification
class SpockDemoSpec extends Specification {
String a
String b
#Shared
String c
def setup() {
println "This method is executed before each specification"
a = "A value"
b = "B value"
}
def setupSpec() {
println "This method is executed only one time before all other specifications"
c = "C value"
}
def "A empty"() {
setup:
a = ""
when:
def result = doSomething(a)
then:
result == ""
}
def "A"() {
when:
def result = doSomething(a)
then:
result == "A VALUE"
}
def "A NullPointerException"() {
setup:
a = null
when:
def result = doSomething(a)
then:
thrown NullPointerException
}
def "C"() {
when:
def result = doSomething(c)
then:
result == 'C VALUE'
}
private String doSomething(String str) {
return str.toUpperCase()
}
}
In this example a and b are set up before each test and c is set up only once before any specification is executed (that's why it requires #Shared annotation to keep it's value between executions).
When we run this test we will see following output in the console:
This method is executed only one time before all other specifications
This method is executed before each specification
This method is executed before each specification
This method is executed before each specification
This method is executed before each specification
The general rule of thumb is that you shouldn't change c value after it is set up in setupSpec method - mostly because you don't know if which order tests are going to be executed (in case you need to guarantee the order you can annotate your class with #Stepwise annotation - Spock will run all cases in order they are defined in this case). In case of a and b you can override then at the single method level - they will get reinitialized before another test case is executed. Hope it helps.

Authentication test running strange

I've just tried to write a simple test for Auth:
use Mockery as m;
...
public function testHomeWhenUserIsNotAuthenticatedThenRedirectToWelcome() {
$auth = m::mock('Illuminate\Auth\AuthManager');
$auth->shouldReceive('guest')->once()->andReturn(true);
$this->call('GET', '/');
$this->assertRedirectedToRoute('general.welcome');
}
public function testHomeWhenUserIsAuthenticatedThenRedirectToDashboard() {
$auth = m::mock('Illuminate\Auth\AuthManager');
$auth->shouldReceive('guest')->once()->andReturn(false);
$this->call('GET', '/');
$this->assertRedirectedToRoute('dashboard.overview');
}
This is the code:
public function getHome() {
if(Auth::guest()) {
return Redirect::route('general.welcome');
}
return Redirect::route('dashboard.overview');
}
When I run, I've got the following error:
EF.....
Time: 265 ms, Memory: 13.00Mb
There was 1 error:
1) PagesControllerTest::testHomeWhenUserIsNotAuthenticatedThenRedirectToWelcome
Mockery\Exception\InvalidCountException: Method guest() from Mockery_0_Illuminate_Auth_AuthManager should be called
exactly 1 times but called 0 times.
—
There was 1 failure:
1) PagesControllerTest::testHomeWhenUserIsAuthenticatedThenRedirectToDashboard
Failed asserting that two strings are equal.
--- Expected
+++ Actual
## ##
-'http://localhost/dashboard/overview'
+'http://localhost/welcome'
My questions are:
Two similar test cases but why the error output differs? First one the mock Auth::guest() is not called while the second one seems to be called.
On the second test case, why does it fail?
Is there any way to write better tests for my code above? Or even better code to test.
Above test cases, I use Mockery to mock the AuthManager, but if I use the facade Auth::shoudReceive()->once()->andReturn(), then it works eventually. Is there any different between Mockery and Auth::mock facade here?
Thanks.
You're actually mocking a new instance of the Illuminate\Auth\AuthManager and not accessing the Auth facade that is being utilized by your function getHome(). Ergo, your mock instance will never get called. (Standard disclaimer that none of the following code is tested.)
Try this:
public function testHomeWhenUserIsNotAuthenticatedThenRedirectToWelcome() {
Auth::shouldReceive('guest')->once()->andReturn(true);
$this->call('GET', '/');
$this->assertRedirectedToRoute('general.welcome');
}
public function testHomeWhenUserIsAuthenticatedThenRedirectToDashboard() {
Auth::shouldReceive('guest')->once()->andReturn(false);
$this->call('GET', '/');
$this->assertRedirectedToRoute('dashboard.overview');
}
If you check out Illuminate\Support\Facades\Facade, you'll see that it takes care of mocking for you. If you really wanted to do it the way that you were doing it (creating an instance of mock instance of Auth), you'd have to somehow inject it into the code under test. I believe that it could be done with something like this assuming that you extend from the TestCase class provided by laravel:
public function testHomeWhenUserIsNotAuthenticatedThenRedirectToWelcome() {
$this->app['auth'] = $auth = m::mock('Illuminate\Auth\AuthManager');
// above line will swap out the 'auth' facade with your facade.
$auth->shouldReceive('guest')->once()->andReturn(true);
$this->call('GET', '/');
$this->assertRedirectedToRoute('general.welcome');
}

Difference between Mock / Stub / Spy in Spock test framework

I don't understand the difference between Mock, Stub, and Spy in Spock testing and the tutorials I have been looking at online don't explain them in detail.
Attention: I am going to oversimplify and maybe even slightly falsify in the upcoming paragraphs. For more detailed info see Martin Fowler's website.
A mock is a dummy class replacing a real one, returning something like null or 0 for each method call. You use a mock if you need a dummy instance of a complex class which would otherwise use external resources like network connections, files or databases or maybe use dozens of other objects. The advantage of mocks is that you can isolate the class under test from the rest of the system.
A stub is also a dummy class providing some more specific, prepared or pre-recorded, replayed results to certain requests under test. You could say a stub is a fancy mock. In Spock you will often read about stub methods.
A spy is kind of a hybrid between real object and stub, i.e. it is basically the real object with some (not all) methods shadowed by stub methods. Non-stubbed methods are just routed through to the original object. This way you can have original behaviour for "cheap" or trivial methods and fake behaviour for "expensive" or complex methods.
Update 2017-02-06: Actually user mikhail's answer is more specific to Spock than my original one above. So within the scope of Spock, what he describes is correct, but that does not falsify my general answer:
A stub is concerned with simulating specific behaviour. In Spock this is all a stub can do, so it is kind of the simplest thing.
A mock is concerned with standing in for a (possibly expensive) real object, providing no-op answers for all method calls. In this regard, a mock is simpler than a stub. But in Spock, a mock can also stub method results, i.e. be both a mock and a stub. Furthermore, in Spock we can count how often specific mock methods with certain parameters have been called during a test.
A spy always wraps a real object and by default routes all method calls to the original object, also passing through the original results. Method call counting also works for spies. In Spock, a spy can also modify the behaviour of the original object, manipulating method call parameters and/or results or blocking the original methods from being called at all.
Now here is an executable example test, demonstrating what is possible and what is not. It is a bit more instructive than mikhail's snippets. Many thanks to him for inspiring me to improve my own answer! :-)
package de.scrum_master.stackoverflow
import org.spockframework.mock.TooFewInvocationsError
import org.spockframework.runtime.InvalidSpecException
import spock.lang.FailsWith
import spock.lang.Specification
class MockStubSpyTest extends Specification {
static class Publisher {
List<Subscriber> subscribers = new ArrayList<>()
void addSubscriber(Subscriber subscriber) {
subscribers.add(subscriber)
}
void send(String message) {
for (Subscriber subscriber : subscribers)
subscriber.receive(message);
}
}
static interface Subscriber {
String receive(String message)
}
static class MySubscriber implements Subscriber {
#Override
String receive(String message) {
if (message ==~ /[A-Za-z ]+/)
return "ok"
return "uh-oh"
}
}
Subscriber realSubscriber1 = new MySubscriber()
Subscriber realSubscriber2 = new MySubscriber()
Publisher publisher = new Publisher(subscribers: [realSubscriber1, realSubscriber2])
def "Real objects can be tested normally"() {
expect:
realSubscriber1.receive("Hello subscribers") == "ok"
realSubscriber1.receive("Anyone there?") == "uh-oh"
}
#FailsWith(TooFewInvocationsError)
def "Real objects cannot have interactions"() {
when:
publisher.send("Hello subscribers")
publisher.send("Anyone there?")
then:
2 * realSubscriber1.receive(_)
}
def "Stubs can simulate behaviour"() {
given:
def stubSubscriber = Stub(Subscriber) {
receive(_) >>> ["hey", "ho"]
}
expect:
stubSubscriber.receive("Hello subscribers") == "hey"
stubSubscriber.receive("Anyone there?") == "ho"
stubSubscriber.receive("What else?") == "ho"
}
#FailsWith(InvalidSpecException)
def "Stubs cannot have interactions"() {
given: "stubbed subscriber registered with publisher"
def stubSubscriber = Stub(Subscriber) {
receive(_) >> "hey"
}
publisher.addSubscriber(stubSubscriber)
when:
publisher.send("Hello subscribers")
publisher.send("Anyone there?")
then:
2 * stubSubscriber.receive(_)
}
def "Mocks can simulate behaviour and have interactions"() {
given:
def mockSubscriber = Mock(Subscriber) {
3 * receive(_) >>> ["hey", "ho"]
}
publisher.addSubscriber(mockSubscriber)
when:
publisher.send("Hello subscribers")
publisher.send("Anyone there?")
then: "check interactions"
1 * mockSubscriber.receive("Hello subscribers")
1 * mockSubscriber.receive("Anyone there?")
and: "check behaviour exactly 3 times"
mockSubscriber.receive("foo") == "hey"
mockSubscriber.receive("bar") == "ho"
mockSubscriber.receive("zot") == "ho"
}
def "Spies can have interactions"() {
given:
def spySubscriber = Spy(MySubscriber)
publisher.addSubscriber(spySubscriber)
when:
publisher.send("Hello subscribers")
publisher.send("Anyone there?")
then: "check interactions"
1 * spySubscriber.receive("Hello subscribers")
1 * spySubscriber.receive("Anyone there?")
and: "check behaviour for real object (a spy is not a mock!)"
spySubscriber.receive("Hello subscribers") == "ok"
spySubscriber.receive("Anyone there?") == "uh-oh"
}
def "Spies can modify behaviour and have interactions"() {
given:
def spyPublisher = Spy(Publisher) {
send(_) >> { String message -> callRealMethodWithArgs("#" + message) }
}
def mockSubscriber = Mock(MySubscriber)
spyPublisher.addSubscriber(mockSubscriber)
when:
spyPublisher.send("Hello subscribers")
spyPublisher.send("Anyone there?")
then: "check interactions"
1 * mockSubscriber.receive("#Hello subscribers")
1 * mockSubscriber.receive("#Anyone there?")
}
}
Try it in the Groovy Web Console.
The question was in the context of the Spock framework and I don't believe the current answers take this into account.
Based on Spock docs (examples customized, my own wording added):
Stub: Used to make collaborators respond to method calls in a certain way. When stubbing a method, you don’t care if and how many times the method is going to be called; you just want it to return some value, or perform some side effect, whenever it gets called.
subscriber.receive(_) >> "ok" // subscriber is a Stub()
Mock: Used to describe interactions between the object under specification and its collaborators.
def "should send message to subscriber"() {
when:
publisher.send("hello")
then:
1 * subscriber.receive("hello") // subscriber is a Mock()
}
A Mock can act as a Mock and a Stub:
1 * subscriber.receive("message1") >> "ok" // subscriber is a Mock()
Spy: Is always based on a real object with original methods that do real things. Can be used like a Stub to change return values of select methods. Can be used like a Mock to describe interactions.
def subscriber = Spy(SubscriberImpl, constructorArgs: ["Fred"])
def "should send message to subscriber"() {
when:
publisher.send("hello")
then:
1 * subscriber.receive("message1") >> "ok" // subscriber is a Spy(), used as a Mock an Stub
}
def "should send message to subscriber (actually handle 'receive')"() {
when:
publisher.send("hello")
then:
1 * subscriber.receive("message1") // subscriber is a Spy(), used as a Mock, uses real 'receive' function
}
Summary:
A Stub() is a Stub.
A Mock() is a Stub and Mock.
A Spy() is a Stub, Mock and Spy.
Avoid using Mock() if Stub() is sufficient.
Avoid using Spy() if you can, having to do so could be a smell and hints at incorrect test or incorrect design of object under test.
In simple terms:
Mock: You mock a type and on the fly you get an object created. Methods in this mock object returns the default values of return type.
Stub: You create a stub class where methods are redefined with definition as per your requirement. Ex: In real object method you call and external api and return the username against and id. In stubbed object method you return some dummy name.
Spy: You create one real object and then you spy it. Now you can mock some methods and chose not to do so for some.
One usage difference is you can not mock method level objects. whereas you can create a default object in method and then spy on it to get the desired behavior of methods in spied object.
Stubs are really only to facilitate the unit test, they are not part of the test. Mocks, are part of the test, part of the verification, part of the pass / fail.
So, say you have a method that takes in a object as a parameter. You never do anything which changes this parameter in the test. You simply read a value from it. That's a stub.
If you change anything, or need to verify some sort of interaction with the object, then it it is a mock.

Grails integration test failing with MissingMethodException

I'm attempting to test a typical controller flow for user login. There are extended relations, as with most login systems, and the Grails documentation is completely useless. It doesn't have a single example that is actually real-world relevant for typical usage and is a feature complete example.
my test looks like this:
#TestFor(UserController)
class UserControllerTests extends GroovyTestCase {
void testLogin() {
params.login = [email: "test1#example.com", password: "123"]
controller.login()
assert "/user/main" == response.redirectUrl
}
}
The controller does:
def login() {
if (!params.login) {
return
}
println("Email " + params.login.email)
Person p = Person.findByEmail(params?.login?.email)
...
}
which fails with:
groovy.lang.MissingMethodException: No signature of method: immigration.Person.methodMissing() is applicable for argument types: () values: []
The correct data is shown in the println, but the method fails to be called.
The test suite cannot use mocks overall because there are database triggers that get called, the result of which will need to be tested.
The bizarre thing is that if I call the method directly in the test, it works, but calling it in the controller doesn't.
For an update: I regenerated the test directly from the grails command, and it added the #Test annotation:
#TestFor(UserController)
class UserControllerTests extends GroovyTestCase {
#Test
void testLogin() {
params.login = [email: "test1#example.com", password: "123"]
Person.findByEmail(params.login.email)
controller.login()
}
}
This works if I run it with
grail test-app -integration UserController
though the result isn't populated correctly - the response is empty, flash.message is null even though it should have a value, redirectedUrl is null, content body is empty, and so is view.
If I remove the #TestFor annotation, it doesn't work even in the slightest. It fails telling me that 'params' doesn't exist.
In another test, I have two methods. The first method runs, finds Person.findAllByEmail(), then the second method runs and can't find Person.findAllByEmail and crashes with a similar error - method missing.
In another weird update - it looks like the response object is sending back a redirect, but to the application baseUrl, not to the user controller at all.
Integration tests shouldn't use #TestFor. You need to create an instance of the controller in your test, and set params in that:
class UserControllerTests extends GroovyTestCase {
void testLogin() {
def controller = new UserController()
controller.params.login = [email:'test1#example.com', password:'123']
controller.login()
assert "/user/main" == controller.response.redirectedUrl
}
}
Details are in the user guide.
The TestFor annotation is used only in unit tests, since this mocks the structure. In the integration tests you have access of the full Grails environment, so there's no need for this mocks. So just remove the annotation and should work.
class UserControllerTests extends GroovyTestCase {
...
}