When I try to run multiple UI tests at the same time in the new Xcode beta, it fails after the first test, with the error "UI Testing Failure: App state is still not terminated" for each test after the first. Anyone got a fix for this?
I have faced the same issue. It seems that, at least in my case, application had never been terminated.
I solved it by putting the following statement in the setUp() method:
continueAfterFailure = false
This should stop a running test process after first failure.
I cannot post a comment, so I will try to answer your question while getting some clarification.
I have faced this issue when running on physical devices. On simulator I did not face this issue. So are you facing this issue when running on device or simulator or both?
If its a device, then there is a known bug which has been reported to Apple. To overcome this issue, at runtime I have inserted dummy test cases between two genuine test cases.
Did you include a
func tearDown() {
super.tearDown()
}
function?
It's needed to terminate the app state after each test before it is re-initialized by the
func setUp() {
super.setUp();
XCUIApplication().launch
}
Related
The process for writing multiple windows tests described in TestCafe documentation seems pretty straightforward:
await t.openWindow('https://url.com/addnewproperty')
or even
const initialWindow = await t.getCurrentWindow()
should do it. However, every time I use any of the Window Management methods I get the errors:
TypeError: t.openWindow is not a function and cannot do anything about it.
Does anyone know what am I doing wrong and how to solve the issue?
TestCafe version 1.9.4
The problem was in different versions of Test Cafe used between main codebase and one of the underlying packages. Alex Kamaev answer was a correct one
We have a react-native app to which we have just added in-app purchases using the expo-in-app-purchases package. Everything seems to be working when testing development builds and TestFlight-distributed release builds on our own devices.
However, we find that the app is consistently not working when reviewed by App Store reviewers.
In the latest review, via our error logging mechanism we discovered that the following error was thrown:
"Must wait for promise to resolve before recalling function."
This was thrown because during review the reviewer caused the following code to be executed twice, 7 minutes apart:
import { getProductsAsync } from 'expo-in-app-purchases';
...
const { responseCode, results } = await getProductsAsync(iOSProductIdArray));
It turns out the initial call to getProductsAsync never resolved, meaning when we called it again, some logic in expo-in-app-purchases (linked above) threw an error.
But what we don't know is why that method doesn't resolve when Apple reviewers use our app, when it always works when we do. The reviewers are signing in to the iTunes Store using the same sandbox credentials we use, and using the same build that we use, but are never able to load product details, which means they can never view purchase options in-app.
Is there something special about the review environment that prevents this function from working? We have theorised that InAppPurchases.connectAsync(), which we call when the app first boots, is failing for reviewers, but we have no idea how or why. Does anyone know how the review environment or process could cause this?
I have some test cases, and also have a test where i want to test that CB get triggered. But every time i run the tests, sometimes it fails, some times it passed. I understand that i need to reset hystrix, but how can i do that? Thanx
I've had some issues with this myself.
Using Hystrix.reset() by itself didn't work when resetting Circuit Breakers.
The following method is working for me.
private void resetCircuitBreakers() {
Hystrix.reset();
HealthCountsStream.reset();
}
We are facing an issue while running karma test cases with phantomJs our phantomJs crashes and gets disconnected.
Is that due to memory leakage or some other issue.Kindly let me know if some one has some suitable solution.
I found that the workaround is to break test cases into multiple grunt task but since we have a lot of test cases more than 1500 so that would not be a feasible task.
We are using the below versions
Node:- 0.10.32
Karma:- 0.12.24
PhantomJs:- 1.9.8 (karma-phantomJs-Launcher)
Please let me know the solutions asap.
There are two reasons I found that this can happen.
PhantomJS does not release memory until its tab is closed so if your test suite is too large, you could be running out of memory.
karma-phantomjs-launcher & karma-phantomjs2-launcher do not hook the stdout/stderr output for their started browser process and so I've seen some instances that the started browser just hangs and gets disconnected, most likely due to its stderr output getting filled up
The first problem can be worked around by splitting your test suite into smaller ones. Or, you could research if there is perhaps a way to tell PhantomJS to run its JavaScript garbage collection, but I have not gone down that road so can't provide much more detail there.
The second problem can be fixed by:
using the latest karma-phantomjs-launcher version that hooks browser the stdout/stderr output (fixed in version 0.2.1)
using a version of karma-phantomjs2-launcher from its pull request #5 which brings in upstream changes from the base karma-phantomJS-launcher project and thus resolves the problem here as well.
I had the same kind of issue with handling random crashes. Though i did not find a way to avoid them, there is the possibility to restart the grunt-task upon a crash.
grunt.registerTask('karma-with-retry', function (opt) {
var done = this.async();
var count = 0;
var retry = function () {
grunt.util.spawn({
cmd : "grunt",
args : ["connect", "karma"], // your tasks
opts: {
stdio: 'inherit'
}
}, function (error, result, code) {
count++;
if (error && code === 90 /* Replace with code thrown by karma */) {
if(count < 5) {
grunt.log.writeln("Retrying karma tests upon error: " + code );
retry();
} else {
done(false);
}
} else {
done(result);
}
});
}
retry();
});
Source https://github.com/ariya/phantomjs/issues/12325#issuecomment-56246505
I was getting Phantom crashed when asserting the following line
dom.should.be.instanceof(HTMLCollection);
Worked on chrome, but phantom was crashing without any useful error message.
I've been able to see the real error message after running the same test on PhantomJS_debug browser with debug option set to true.
The following error message started showing up.
The instanceof assertion needs a constructor but object was given.
Instead of
PhantomJS has crashed. Please read the bug reporting guide at
<http://phantomjs.org/bug-reporting.html> and file a bug report.
So chrome was ok with the assertion but phantom 2.1.1 is crashing with the above error. Hope this will help.
I have seen people posting about this here and elsewhere, but I haven't found any solution that works. I am using XCode 4.4 and have a bunch of unit tests set up. I have ran them all before on this project, so I know that they do pass/fail when they are supposed to if they are actually ran.
I have about 15 test suites, and each one contains 1-7 tests. On most attempts, all of the test suites finished (and passed) except for 1 (FooTests). It gives the warning:
FooTests did not finish
testFoo did not finish
XCode will report that testing was successful, regardless of what happens in unfinished tests. Another thing to note, sometimes it is a different test that will not finish, and sometimes multiple suites will not finish. I have not noticed a case where all tests do finish, but judging by this seemingly random behaviour I believe that it is possible.
So, is this a bug in XCode? I can't think of any other reason that tests randomly don't finish and then cause XCode to report that everything was successful. Are there any solutions?
I am on XCode 4.5.2. For application unit test, if your test suites finish so quick that the main application is not correctly loaded before that, you will get the warning. You can simply avoid the problem by adding a sleep at the end of your test like following. It doesn't happen for logic unit test.
[NSThread sleepForTimeInterval:1.0];
I've just had this problem with XC4.5DP4.
I had a test which does some work in a loop, and does nothing else when it falls out of the loop, and I was getting the "did not finish" error.
In an attempt to prove that the test was finishing, I added this as the very last line:
NSLog(#"done");
Not only does "done" get printed to the output - now Xcode says that the test has finished.
Seems to be a workaround... go figure.
I'm using XCode46-DP3 and I've just resolved this problem in my tests. I've several tests that start a web server and then execute http call to it; at the end of the test the web server is stopped. In last week these tests have begun to have the warning 'did not finish'.
For me has been enough to add the following sleep at the end of these tests (precisely I've added it in the tearDown):
- (void)tearDown {
[self.httpServer stop];
[NSThread sleepForTimeInterval:1.0];
self.httpServer = nil;
self.urlComposer = nil;
}
The problem seems that your tests terminate too quickly for Xcode to receive and parse the log messages that indicate failure or success. Sleeping for 1 second in the last test case run worked reliably for me, where 0.0 didn't. To see which test case is the last test case, check the test action in the Scheme dialog.
I created a separate WorkaroundForTestsFinishingTooFast test case, with a single method:
- (void)testThatMakesSureWeDontFinishTooFast
{
[NSThread sleepForTimeInterval:1.0];
}
The problem is that as you add more test cases, you'll have to ensure that this test is the last method run. That means removing and adding this class from the Test action as reordering of test cases is not allowed. On the other hand, you're only delaying your entire test bundle by 1 second.
I had the same warnings in the Log Navigator. I fixed it for me.
In my project I have got 2 Schemes, one for running the project and one for the unit tests.
Go to Product --> Edit Scheme...
Select the UnitTest Scheme in the scheme picker
Select the "Test"-Icon on the Left
Change the debugger from LLDB to GDB and press OK
Tests should finish now. (For me it worked fine)
For me the solution was to slim down the logging output from the parts of the app that the tests were testing. I think xcode couldn't parse the tests output in time because of the other output I had throughout the app.
I had the same problem running with XCode 4.6, the reason for it, in my case, was inconsistency between the scheme and the actual unit tests in the test suits.
In the scheme I had some suits checked but in their .m file some unit tests were commented.
To solve the problem: either uncomment the test or deselected the file/suit in the scheme and all shall became green again :)
for people like me that forgot how to reach the scheme these are the required steps:
right click on the project name in the scheme section (right to the stop button)
choose edit scheme
choose the test debug
click on the triangle next to the unit test project and next to each file you have a check box
uncheck files that you placed unit test in comments
hope this helps