I have a series of tests that are dependent on a step in the middle (such as creating an account). The API that I'm using for this is a kind of brittle (which is a separate problem), and sometimes fails. I'd like to be able to just quit the tests in the middle there when that fails, instead of wait for TestCafe to fail the initial assertions for the next few tests that follow. Is there a way to get the test controller to stop, or signify to the fixture that the tests should stop? I immediately thought of Spock's #Stepwise annotation, but I can't find anything like that in the TestCafe docs.
The Stop on First Fail option stops the entire run once a failed test occurred. If I understand your scenario correctly, you could add an assertion for a successful account creation and if it fails, exit the entire run with this option.
CLI Documentation
API Documentation (under Parameters)
I am running some jasmine tests on a function that is logging something. Each time I run the tests I see that log in the output of the tests. I have quite a few logs in my functions that I am testing and didn't see a way to suppress the logs in the jasmine output.
My actual tests are spying to make sure that console.log is being called with the correct string.
Suppressing the logs in the jasmine output is really more for testing aesthetics (I just like to see a nice clean green passing and not all the logs).
If you are running your tests with karma, edit your karma.config.js and add:
client: {
captureConsole: false
}
You can put spy on the console method and expect it to have been called, This is how I am using it in my Jasmine Unit test cases. Hope it helps(replace 'warn' with 'log')
spyOn(console, 'warn');
fixture.detectChanges();
component.doSomething(dummyEventObj);
fixture.detectChanges();
expect(console.warn).toHaveBeenCalled();
Make sure your spy isn't calling the real console.log(). Something like this should do the trick spyOn(console, 'log');.
I wrote a script in selenium with TestNg framework such that the failed test method will be rerun, and this is at Test Level by indicating the location of the class that implements the IRetryAnalyzer in #Test()
TestNG had successfully rerun the failed test method; the output has the following information: for the first run, Failed Test Method is marked as Failed, and after rerunning it, the method is marked as skipped. What does it mean? Could anyone help me to interpret the result?
Here is the screenshot of the result:
There is an ongoing issue on the official project.
You're welcome to follow:
https://github.com/cbeust/testng/issues/878
Is there a way to stop test execution if assertion failed so other test group will not be executed ?
From the documentation:
Nightwatch.js instance has two objects containing the same methods to perform
assertions on elements:
.assert - when an assertion fails, the test ends, skipping all other assertions.
.verify - when an assertion fails, the test logs the failure and continues with
other assertions.
So I guess use assert? Unless you are and that's not working for you?
Using assert will stop the current test, but to stop the whole suite, you need to set the setting 'skip_testcases_on_fail' to true in your nightwatch.json config file.
I have seen people posting about this here and elsewhere, but I haven't found any solution that works. I am using XCode 4.4 and have a bunch of unit tests set up. I have ran them all before on this project, so I know that they do pass/fail when they are supposed to if they are actually ran.
I have about 15 test suites, and each one contains 1-7 tests. On most attempts, all of the test suites finished (and passed) except for 1 (FooTests). It gives the warning:
FooTests did not finish
testFoo did not finish
XCode will report that testing was successful, regardless of what happens in unfinished tests. Another thing to note, sometimes it is a different test that will not finish, and sometimes multiple suites will not finish. I have not noticed a case where all tests do finish, but judging by this seemingly random behaviour I believe that it is possible.
So, is this a bug in XCode? I can't think of any other reason that tests randomly don't finish and then cause XCode to report that everything was successful. Are there any solutions?
I am on XCode 4.5.2. For application unit test, if your test suites finish so quick that the main application is not correctly loaded before that, you will get the warning. You can simply avoid the problem by adding a sleep at the end of your test like following. It doesn't happen for logic unit test.
[NSThread sleepForTimeInterval:1.0];
I've just had this problem with XC4.5DP4.
I had a test which does some work in a loop, and does nothing else when it falls out of the loop, and I was getting the "did not finish" error.
In an attempt to prove that the test was finishing, I added this as the very last line:
NSLog(#"done");
Not only does "done" get printed to the output - now Xcode says that the test has finished.
Seems to be a workaround... go figure.
I'm using XCode46-DP3 and I've just resolved this problem in my tests. I've several tests that start a web server and then execute http call to it; at the end of the test the web server is stopped. In last week these tests have begun to have the warning 'did not finish'.
For me has been enough to add the following sleep at the end of these tests (precisely I've added it in the tearDown):
- (void)tearDown {
[self.httpServer stop];
[NSThread sleepForTimeInterval:1.0];
self.httpServer = nil;
self.urlComposer = nil;
}
The problem seems that your tests terminate too quickly for Xcode to receive and parse the log messages that indicate failure or success. Sleeping for 1 second in the last test case run worked reliably for me, where 0.0 didn't. To see which test case is the last test case, check the test action in the Scheme dialog.
I created a separate WorkaroundForTestsFinishingTooFast test case, with a single method:
- (void)testThatMakesSureWeDontFinishTooFast
{
[NSThread sleepForTimeInterval:1.0];
}
The problem is that as you add more test cases, you'll have to ensure that this test is the last method run. That means removing and adding this class from the Test action as reordering of test cases is not allowed. On the other hand, you're only delaying your entire test bundle by 1 second.
I had the same warnings in the Log Navigator. I fixed it for me.
In my project I have got 2 Schemes, one for running the project and one for the unit tests.
Go to Product --> Edit Scheme...
Select the UnitTest Scheme in the scheme picker
Select the "Test"-Icon on the Left
Change the debugger from LLDB to GDB and press OK
Tests should finish now. (For me it worked fine)
For me the solution was to slim down the logging output from the parts of the app that the tests were testing. I think xcode couldn't parse the tests output in time because of the other output I had throughout the app.
I had the same problem running with XCode 4.6, the reason for it, in my case, was inconsistency between the scheme and the actual unit tests in the test suits.
In the scheme I had some suits checked but in their .m file some unit tests were commented.
To solve the problem: either uncomment the test or deselected the file/suit in the scheme and all shall became green again :)
for people like me that forgot how to reach the scheme these are the required steps:
right click on the project name in the scheme section (right to the stop button)
choose edit scheme
choose the test debug
click on the triangle next to the unit test project and next to each file you have a check box
uncheck files that you placed unit test in comments
hope this helps