Let me describe a simple use-case:
Running all tests in our project may take up to 10 minutes.
Sometimes I see an obvious bug in my code after the first failed test, so I want to stop running all tests, fix the bug and re-run them. Unfortunately, I can either re-run all tests from the beginning, or re-run failed tests only.
Is there a plugin for IDEA which allows me to re-run failed tests AND tests, which weren't yet executed when I pressed "STOP"?
Atlassian has the solution for your problem: Clover. But it is commercial.
This goes against the idea of a test suite. Normally you want to run all your tests specifically so you know you haven't broken anything somewhere unexpected. If you change the code and then run a subset of the tests, the possibility exists that you broke something and one of the skipped tests would have failed. This is a case of not getting your cake and eating it too.
If you find a bug in an early test, by all means stop the suite. Fix the bug but then run the suite from the beginning.
Related
We have currently about 200 test features. We start to face something strange, most of the times tests are just stuck and would not proceed when we run mvn test command as the following:
mvn clean test -Dcucumber.options="--tags $tags" -Dtest=TestRunner -Dkarate.env=$env
Some tests would run as it was perfectly fine. But at some point the rest will just stuck as it it hangs.
We run the tests in parallel using 10 threads.
It stucks like this
Anybody experienced similar things? Any ideas what could possibly went wrong?
Thanks
This should be fixed in 0.9.5.RC3 - it is stable to use for API testing, so I recommend you upgrade.
If anyone faces this problem for any other version of Karate, please understand that the best (and possibly only) way to troubleshoot or solve this - is to follow this process: https://github.com/karatelabs/karate/wiki/How-to-Submit-an-Issue
I actually have the same problem as you but I can't comment because of reputation, my project works with Gradle and I'm using IntelliJ IDEA and JDK1.8(at another moment before all this I tried Jetbrains SDK11 but had the same problem, I downgraded to java 8 and everything worked again) on this ocassion I did as peter said and upgraded to 0.9.5.RC4 but still when I execute some of my features they never end, for example, I'm currently working on a very simple feature that calls another feature for login, it works for many other features but for this one it appears to get to the end of its execution and never go back to the caller feature, as I was running out of options I made a new simple project copied the resources folder I store my features in and my run parallel class and tried again but it behaves in the same way, the execution never ends.
I'll upload an image with my screen while it executes as you can see it's been executing for 15 minutes
projectView
I am working with TestCafé 1.8.1 and testcafe-browser-provider-electron 0.0.14. Let's say that an issue in the application force it to get stuck and hang one test execution, and for that reason, the rest of the test in my suite cannot continue executing. Is there a way to force TestCafé to abort the execution of that test after a timeout and continue running the rest of the test in the suite?
I have faced this issue several times and it's a problem because I am not able to see the results of the rest of the "good" tests, just because of one test that hanged the whole execution.
TestCafe reloads a browser and restarts the latest test if the application got stuck. Currently TestCafe cannot drop this test in such a case.
We have currently about 200 test features. We start to face something strange, most of the times tests are just stuck and would not proceed when we run mvn test command as the following:
mvn clean test -Dcucumber.options="--tags $tags" -Dtest=TestRunner -Dkarate.env=$env
Some tests would run as it was perfectly fine. But at some point the rest will just stuck as it it hangs.
We run the tests in parallel using 10 threads.
It stucks like this
Anybody experienced similar things? Any ideas what could possibly went wrong?
Thanks
This should be fixed in 0.9.5.RC3 - it is stable to use for API testing, so I recommend you upgrade.
If anyone faces this problem for any other version of Karate, please understand that the best (and possibly only) way to troubleshoot or solve this - is to follow this process: https://github.com/karatelabs/karate/wiki/How-to-Submit-an-Issue
I actually have the same problem as you but I can't comment because of reputation, my project works with Gradle and I'm using IntelliJ IDEA and JDK1.8(at another moment before all this I tried Jetbrains SDK11 but had the same problem, I downgraded to java 8 and everything worked again) on this ocassion I did as peter said and upgraded to 0.9.5.RC4 but still when I execute some of my features they never end, for example, I'm currently working on a very simple feature that calls another feature for login, it works for many other features but for this one it appears to get to the end of its execution and never go back to the caller feature, as I was running out of options I made a new simple project copied the resources folder I store my features in and my run parallel class and tried again but it behaves in the same way, the execution never ends.
I'll upload an image with my screen while it executes as you can see it's been executing for 15 minutes
projectView
Whenever I run UI Tests locally, they pass, and I can watch the simulator do exactly what it's supposed to.
Then, when the tests run on Jenkins, they fail on the first line.
If it matters, this is the failure:
UI Testing Failure - No matches found for Table
The above error never happens when the tests are running locally.
I appreciate all of the help, and also want to help others with the same issue as much as possible. It's tough, because I can't post code or give too much information, since the code I'm working on is proprietary.
Here was the issue:
I didn't realize that Jenkins just ran the tests on a simulator, on a computer, just like I would do on mine. I had to handle logging into the app in Jenkins' simulator, so that all of the XCUIElements were accessible.
The tests weren't passing the first line, because I wanted to tap a "Settings" button, and a user wasn't even logged in.
Please comment if I can clarify or explain further.
I'm running nunit tests externally via Nunit3-console.
I'm not able to see any console.logs/console.Writeln
I find it mandatory to be able to track in real time, every step of the test.
I've read that nunit3 framework created a parallel test run feature, which is why the real time test output logs have been taken off.
But if I want to enjoy both worlds?
How can I trigger console logs during a test run?
Thanks in advance.
Nunit3 framework doesn't support live hard coded console outputs when running tests from Nunit3-console, because nunit3 framework was meant to bring the parallel testing methodology, they found it pointless to allow such outputs when more than 1 test is running in parallel.
I solved this by downgrading to Nunit 2.6.4 which isnt supporting parallel testing and allows me to fire console outputs from my test.