I am having an issue in Github Actions.
I am using Cypress to test the frontend of my app, and everything is working perfectly in the Cypress app.
But when I am pushing everything on master in GitHub and run the test via Github Actions, every now and then, the same flaky test is failing with he following error.
Timed out retrying after 5000ms: Expected to find element "xxx" but never found it.
This line is the problematic one:
cy.purposeElement("delete_user_dialog").should('be.visible')
The element we are talking about takes 0.2s to appear with a fade-in animation.
My guess is that the page was slow to respond, while Cypress was fast to act.
How can I avoid solve a flaky test like that?
I could use a cy.wait but it is not recommended and I do not want to increase the time of the test.
Plus, the result is as flaky.
It may be as simple as increasing the timeout, especially if the test works perfectly when running on the local machine.
cy.purposeElement("delete_user_dialog", {timeout:20_000}).should('be.visible')
To be 100% sure, do a burn test which is described here: Burning Tests with cypress-grep
Related
As mentioned in the title, the cypress client proceeds on to other tests without marking a test as failed where it has in fact found a mismatch in the expectation. This behavior can be seen in the attached image:
This is intermittent but can go completely unnoticed when the tests run in a CI environment.
How am I supposed to debug & fix this issue?
Is there a way to tell the system to restart the test in case a specific rare system error comes up?
Basically sometimes we get strange errors related to elements being "obscured" or "stale", but which do not mean the site is not working etc. It has to do with the site's latency I believe like CSS not loading quickly enough etc.
For example is there a directive to tell the system that if an error like
[Facebook\WebDriver\Exception\ElementClickInterceptedException] Element
\<li id="nav_step0" class="nav-steps selected"> is not clickable at point (330,237)
because another element \<div id="ajaxloading_mask" class="mask"> obscures it
To simply relaunch the test again?
No, there is no way to relaunch failed test on specific error.
You can rerun all failed tests:
codecep run || codecept run -g failed
This command executes all tests, if any tests failed, it reruns only failed tests.
When using the APIs defined by Protractor & Jasmine (the default/supported runner for Protractor), the tests will always work okay on individual developer laptops. For some reason when the test runs on the Jenkins CI server, they will fail (despite being in the same docker containers on both hosts, and that was wildly frustrating.)
This error occurs: A Jasmine spec timed out. Resetting the WebDriver Control Flow.
This error also appears: Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
Setting getPageTimeout & allScriptsTimeout to 30 seconds had no effect on this.
I tried changing jasmine.DEFAULT_TIMEOUT_INTERVAL to 60 seconds for all tests in this suite, once the first error appears then every test will wait the full 60 seconds and time out.
I've read and reread Protractor's page on timeouts but none of that seems relevant to this situation.
Even stranger still, it seems like some kind of buffer issue - at first the tests would always fail on a particular spec, and nothing about that spec looked wrong. While debugging I upgraded the selenium docker container from 2.53.1-beryllium to 3.4.0-einsteinium and the tests still failed but they failed a couple specs down - suggesting that maybe there was some optimization in the update and so it was able to get more done before it gave out.
I confirmed that by rearranging the order of the specs - the specs that had failed consistently before were now passing and a test that previously passed began to fail (but around the same time in the test duration as the other failures before the reorder.)
Environment:
protractor - 5.1.2
selenium/standalone-chrome-debug - 3.4.0-einsteinium
docker - 1.12.5
The solution ended up being simple - I first found it on a chrome bug report, and it turned out it was also listed right on the front page of the docker-selenium repo but the text wasn't clear as to what it was for when I'd read it the first time. (It says that selenium will crash without it, but the errors I was getting from Jasmine were just talking about timeouts, and that was quite misleading.)
Chrome apparently utilizes /dev/shm, and apparently that's fairly small in docker. There are workarounds for chrome and firefox linked from their README that explain how to resolve the issue.
I had a couple test suites fail after applying the fix but all the test suites have been running and passing for the last day, so I think that was actually the problem and that this solution works. Hope this helps!
I'm searching for the equivalent of mocha:
mocha test.spec.js -w
opening nigtwatch manually on every change burns a lot of time during test writing.
any ideas of this?
I've checked nightwatch docs and google, nothing came up
From what I've seen, there isn't a --watch mode or command for nightwatch. This is only an answer in that what I've seen confirms that this doesn't seem to exist.
Here are the sources that led me to this conclusion, including possible leads for other solutions:
I've found an issue in the nightwatch repo asking about that in 2016: https://github.com/nightwatchjs/nightwatch/issues/1061. The answer then was
'No, but you can use something like the grunt-contrib-watch module.'
I found a stackoverflow question in 2018 that talks about how nightwatch, and other things using selenium, always return exit code 1: Nightwatch.js always returns exit code 1. That leads me to believe that nightwatch always exits its tests. The question was asking about integrate Nightwatch.js tests in a Jenkins Job. A comment offers a possible solution for that, so maybe it's possible to make a wrapper for nightwatch this way:
This happens with other selenium based libraries as well. I think I ran into this issue with protractor in the past. The way we tackled it was by parsing the output instead of using the exit code and looked for any failures. If none were found then we returned 0 explicitly... you can check for failed tests using browser.currentTest.results.failed and then figure out a way to pass or fail your build based on that.
I'd like to stop or pause Selenium test when running with playframework, because I can't see what is wrong!
Is there a way to preserve the window with my application on the step where it failed? It automatically closes that window!
You could try a few things after you pin point exactly which step causes the error:
Put a wait command in your selenium test to pause the test so you can see what is going on in the UI
You could start the debugger and put a break point on some code that gets executed on the server after the failure happens (assuming the test continues running after the failure)
You could use the selenium test runner panel to slow down the speed of the test so you can see the error (i.e. I'm referring to the speed slider that you see when you run the tests in the browser via the url http://localhost:9000/#tests)