We have e2e test integrated with Jenkins system. For a few weeks this test successfully ran both, locally and from Jenkins (as a part of the build pipeline).
At the end of Sprint, I modified the script to reflect Sprint changes and made sure it passed locally. Then, I merged the changes with master. Now, e2e runs from Jenkins are failing 100% of the time, while when I locally connect to QA envs there is no problem.
The error I am getting is - Element is not clickable at point (x, y) which I cannot reproduce locally.
The server doesn't have a real screen so I cannot go out there and see what's going on. Resolutions are perfectly matching. I have other people running this test locally and there is no problem.
What could possibly cause these failures and how do I troubleshoot this problem?
Thanks for your help!
Its a question from 1000ft and pretty difficult to identify where exactly the issue could be but I listed down some probable causes/debugging tips that could help you
1.Whats your checkout strategy from source code repository? Check job workspace and it should have the most recent code and check if its indeed the latest one.
May be configure Job to always pick a new version instead of 'update'
2.Add a reporter based on the test framework you are using especially the ones which provide screenshots. Refer my blog for more details -
3.Check the stack trace of your error from Jenkins console report and verify the exact trigerring point
Related
We have currently about 200 test features. We start to face something strange, most of the times tests are just stuck and would not proceed when we run mvn test command as the following:
mvn clean test -Dcucumber.options="--tags $tags" -Dtest=TestRunner -Dkarate.env=$env
Some tests would run as it was perfectly fine. But at some point the rest will just stuck as it it hangs.
We run the tests in parallel using 10 threads.
It stucks like this
Anybody experienced similar things? Any ideas what could possibly went wrong?
Thanks
This should be fixed in 0.9.5.RC3 - it is stable to use for API testing, so I recommend you upgrade.
If anyone faces this problem for any other version of Karate, please understand that the best (and possibly only) way to troubleshoot or solve this - is to follow this process: https://github.com/karatelabs/karate/wiki/How-to-Submit-an-Issue
I actually have the same problem as you but I can't comment because of reputation, my project works with Gradle and I'm using IntelliJ IDEA and JDK1.8(at another moment before all this I tried Jetbrains SDK11 but had the same problem, I downgraded to java 8 and everything worked again) on this ocassion I did as peter said and upgraded to 0.9.5.RC4 but still when I execute some of my features they never end, for example, I'm currently working on a very simple feature that calls another feature for login, it works for many other features but for this one it appears to get to the end of its execution and never go back to the caller feature, as I was running out of options I made a new simple project copied the resources folder I store my features in and my run parallel class and tried again but it behaves in the same way, the execution never ends.
I'll upload an image with my screen while it executes as you can see it's been executing for 15 minutes
projectView
I really don't know how to ask question to Google about this, so I excuse me that it is naive.
Our team is developing SPA application in ReactJS. We also do back-end programming for NodeJS. Our project recently got more e2e tests. They are written using webdriver.io packages. Everything works as expected but circa 30 tests run about 50 minutes. It is too long to pause developer work and force him to run tests.
We came with the idea that now when we have so many tests, we need to run them on separate computer (other than a developer's laptop, further I call it e2e-laptop).
So I programmed a bash script and installed Ubuntu on a e2e-laptop. My idea is, that developer who wants to run e2e test logs in on e2e-laptop with ssh, runs specified script with arguments (eg: --rev= specific git revision the tests should run on, --email= where to send Allure report) and logs out. After tests are done he gets Allure report in his mailbox.
This all sounds to me OK, but not very well. It works - it is like a dirty MVP. But what I really would like to give my team is the web browser based UI that gives the features my script has. I can imagine this software is hosted on e2e-laptop, every developer can open its webpage address in his local browser. Then after authorization, there are options: run all specs, run chosen specs, send report and more. It would be the best if that software could also allow simultaneous running of tests commissioned by multiple developers.
What software I need?
You need a continuous integration tool. https://stackify.com/top-continuous-integration-tools/
I recommend Jenkins.
I would first try to run your selenium tests headless in a docker container on your laptop. Once you are able to do that, use that same configuration in your docker container running in Bitbucket pipelines. It could actually be the same container and the same scripts. Then, developers can just make a branch and work with the tests on that branch. If only a certain subset of tests need to run, then the developer can make the necessary changes on his or her local branch to run those tests and push it up to Bitbucket. This should help with the configuration https://github.com/SeleniumHQ/docker-selenium.
Whenever I run UI Tests locally, they pass, and I can watch the simulator do exactly what it's supposed to.
Then, when the tests run on Jenkins, they fail on the first line.
If it matters, this is the failure:
UI Testing Failure - No matches found for Table
The above error never happens when the tests are running locally.
I appreciate all of the help, and also want to help others with the same issue as much as possible. It's tough, because I can't post code or give too much information, since the code I'm working on is proprietary.
Here was the issue:
I didn't realize that Jenkins just ran the tests on a simulator, on a computer, just like I would do on mine. I had to handle logging into the app in Jenkins' simulator, so that all of the XCUIElements were accessible.
The tests weren't passing the first line, because I wanted to tap a "Settings" button, and a user wasn't even logged in.
Please comment if I can clarify or explain further.
I have a Grails application that, when run on my local Windows machine, passes all tests in my integration test suite. When I deploy my app to my Test environment in Jenkins, and run the same suite of tests, a few of them are failing for inexplicable reasons.
I think the Test box is Linux but I am not sure. I am using mocks in my Grails app and am wondering if that may be causing confusion in values returned.
Has anyone any ideas?
EDIT:
My app translates an XML document into a new XML document. One of the elements in the returned XML document is supposed to be PRODUCT but comes back as product.
The place where this element is set is from an in-memory database that is populated from a DB script. It is the same DB script that is used locally and on my Test environment.
The app does not read any config files that would be different in different environments.
Like the others have stated the really isn't enough information here to help give a solid answer. A couple of things that I would look at are:
If it's integration tests that are failing maybe you've got some "bad tests" that are dependent on certain data that does not exist in your test environment that Jenkins is running against.
There is no guaranteed consistency for test execution order across machines/platforms. So it's entirely possible that the tests pass for you locally just because they run in a certain order and leave things mocked out or data setup from one test that is needed in another. I wrote a plugin a while ago (http://grails.org/plugin/random-test-order) to help identify these problems. I haven't updated the plugin since Grails 1.3.7 so it may not work with 2.0+ grails apps.
If the steps above don't identify the problem knowing any differences in how you are invoking the tests on Jenkins vs. Local would be beneficial. For example if you specify a specific grails environment (http://grails.org/doc/latest/guide/conf.html#environments) when running on Jenkins and what the differences are between that and the grails environment used on your local.
While I only have a github repository that I'm pushing to (alone), I often forget to run tests, or forget to commit all relevant files, or rely on objects residing on my local machine. These result in build breaks, but they are only detected by Travis-CI after the erroneous commit. I know TeamCity has a pre-commit testing facility (which relies on the IDE in use), but my question is with regards to the current use of continuous integration as opposed to any one implementation. My question is
Why aren't changes tested on a clean build machine - such as those which Travis-CI uses for post-commit tesing - before those changes are committed?
Such a process would mean that there would never be build breaks, meaning that a fresh environment could pull any commit from the repository and be sure of its success; as such, I don't understand why CI isn't implemented using post-commit testing.
I preface my answer with the details that I am running on GitHub and Jenkins.
Why should a developer have to run all tests locally before committing. Especially in the Git paradigm that is not a requirement. What if, for instance, it takes 15-30 minutes to run all of the tests. Do you really want your developers or you personally sitting around waiting for the tests to run locally before your commit and push your changes?
Our process usually goes like this:
Make changes in local branch.
Run any new tests that you have created.
Commit changes to local branch.
Push local changes remotely to GitHub and create pull request.
Have build process pick up changes and run unit tests.
If tests fail, then fix them in local branch and push them locally.
Get changes code reviewed in pull request.
After approval and all checks have passed, push to master.
Rerun all unit tests.
Push artifact to repository.
Push changes to an environment (ie DEV, QA) and run any integration/functional tests that rely on a full environment.
If you have a cloud then you can push your changes to a new node and only after all environment tests pass reroute the VIP to the new node(s)
Repeat 11 until you have pushed through all pre-prod environments.
If you are practicing continuous deployment then push your changes all the way to PROD if all testing, checks, etc pass.
My point is that it is not a good use of a developers time to run tests locally impeding their progress when you can off-load that work onto a Continuous Integration server and be notified of issues that you need to fix later. Also, some tests simply can't be run until you commit them and deploy the artifact to an environment. If an environment is broken because you don't have a cloud and maybe you only have one server, then fix it locally and push the changes quickly to stabilize the environment.
You can run tests locally if you have to, but this should not be the norm.
As to the multiple developer issue, open source projects have been dealing with that for a long time now. They use forks in GitHub to allow contributors the chance to suggest new fixes and functionality, but this is not really that different from a developer on the team creating a local branch, pushing it remotely, and getting team buy-in via code review before pushing. If someone pushes changes that break your changes then you try to fix them yourself first and then ask for their help. You should be following the principle of "merging early and often" as well as merging in updates from master to your branch periodically.
The assumption that if you write code and it compiles and tests are passed locally, no builds could be broken is wrong. It is only so, if you are the only developer working on that code.
But let's say I change the interface you are using, my code will compile and pass tests
as long as I don't get your updated code That uses my interface.
Your code will compile and pass tests as long as you don't get my update in the interface.
And when we both check in our code, the build machine explodes...
So CI is a process which basically say: put your changes in as soon as possible
and test them in the CI server (it should be of course compiled and tested locally first).
If all developers follow those rules,
the build will still break, but we will know about it sooner rather than later.
The CI server is not the same as the version control system. The CI server, too, checks the code out of the repository. And therefore the code has already been committed when it gets tested on the CI server.
More extensive tests may be run periodically, rather than at time of checking in, on whatever is the current version of the code at the time of testing. Think of multi-platform tests or load tests.
Generally, of course, you'll unit test your code on your development machine before checking it in.