I have a React frontend and a Node backend, I've made several E2E tests with Cypress for the frontend and run them locally. I love how end to end testing allows me to catch errors in the frontend as well as on the backend! so I'd like to have a way to run them automatically when sending a PR.
I'm using bitbucket pipelines, and I've configured it to run the npm test command which works perfectly to run my unit tests, but what I still don't understand is how to be able to run my Cypress tests automatically, because I'd need to have access to the backend repository from my pipeline that runs on the frontend repo.
What I've tried
I've read the documentation and played with the example repo, but I still don't understand how could I automate running my tests, on the example both backend and frontend are on the same repository.
I'm sorry if this is a vague question, I just don't seem to get if this is even possible with bitbucket pipelines, if it's not, what other tool could help me run my Cypress test in a similar way that I do locally? (running both backend and frontend).
I've really tried to search for an answer to this, maybe it's too obvious and I'm just missing something but I don't seem to find anything on the internet about this, any help will be very appreciated!
When your frontend and backend are versioned in different repositories, then you have to check out at least one of the two repositories (e.g. the other for which the pipeline is not currently being executed) during the pipeline execution to get access to the code and thus have the possibility to start frontend and backend together locally to run your tests.
This question has also already been asked and answered here:
https://community.atlassian.com/t5/Bitbucket-questions/Access-multiple-Bitbucket-repositories-from-a-single-Pipeline/qaq-p/1783419
Related
I have vue project which published on Digital Ocean. The main problem is when i make some changes on FileZilla it is not affect on website. How can i solve this issue?
This is not an issue per-se. This is just the way how modern web development works. Vue.js (but also Nuxt) is using a bundler right now (Webpack, Vite are the most common), hence to go to production it needs to be bundled each time you push something to it.
If you upload something via FTP or SSH and edit some source code, a bundle step will be required in order to get any changes on the actual webapp.
Backend languages may not need that, for example you could SSH into a server and change some .php file, if you F5 the page it will be updated in real time. But this is not how frontend JS code works, it needs to be optimised.
Another thing, sending code via SSH/FTP is not really a good workflow because it is not easily trackable, no version-controlled, will not trigger any build flags in case of an error etc...
The best approach is to have a git repo + some build step included in some CI.
A common platform for it is Netlify, you connect a Github repo, you tell which command to use to build the project and each time you push some code, it may do some checks/tests/optimizations/etc... via Github Actions before being released automatically to production (updated on your webapp).
This workflow have a lot of benefits as one may tell but is also de-facto, the official/regular approach for modern Web development on the frontend.
I'm writing an API that consists of several microservices. I have the code in a private Gitlab repo. I have a custom CI/CD pipeline configured to run a couple of different steps automatically on every commit to master (e.g. build, test, deploy to a dev environment). Deploying to prod is manual.
I have written some unit tests around this code, which naturally test only small units of the code. These, of course, are run with every commit, because if they fail, that means something in the code has broken.
I also have regression tests which we run after deploying. One of these is actually a bash script that uses curl to hit my production endpoint with certain parameters and checks to make sure that I'm getting 200 responses. I have parameterized this script so I can easily point it at my dev environment (instead of prod).
I use this regression test (and others like it) to check that my already-deployed service is functioning properly. And I run it right after deploying as a final, double-check to confirm that everything is working. But I want to automate that.
My question is where does this fit in a CI/CD workflow? It wouldn't make sense to run this kind of regression test on a commit, because that commit is not necessarily coupled with a deploy. And because there are any number of reasons why the service might be down that are unrelated to whatever code changes went into the most recent commit. In other words, the pipeline should not fail because of external circumstances.
Are there any best practices for running and automating regressions tests?
Great question. There are a couple of interesting points here.
When to run the regression tests (as they exist today) in your CI / CD environment.
The obvious answer to this is to run as a post deploy step. Using the same approach you are currently using to limit the deploy step to the master branch only you can limit this post deploy step to the master branch only.
If you add more details about your environment. For example the CI / CD system that you are using and your current configuration I would be very happy to provide more concrete details on how to achieve this.
It wouldn't make sense to run this kind of regression test on a commit
An interesting approach that I have seen a couple of times. Is using a cloud service (AWS / GCloud etc.) to spin up an environment on each CI run. This means that the full pipeline can be run for every commit. While it takes more resources, it means that you can find issues prior to merging to master. Of course up to you whether the ROI adds up in your environment.
I really don't know how to ask question to Google about this, so I excuse me that it is naive.
Our team is developing SPA application in ReactJS. We also do back-end programming for NodeJS. Our project recently got more e2e tests. They are written using webdriver.io packages. Everything works as expected but circa 30 tests run about 50 minutes. It is too long to pause developer work and force him to run tests.
We came with the idea that now when we have so many tests, we need to run them on separate computer (other than a developer's laptop, further I call it e2e-laptop).
So I programmed a bash script and installed Ubuntu on a e2e-laptop. My idea is, that developer who wants to run e2e test logs in on e2e-laptop with ssh, runs specified script with arguments (eg: --rev= specific git revision the tests should run on, --email= where to send Allure report) and logs out. After tests are done he gets Allure report in his mailbox.
This all sounds to me OK, but not very well. It works - it is like a dirty MVP. But what I really would like to give my team is the web browser based UI that gives the features my script has. I can imagine this software is hosted on e2e-laptop, every developer can open its webpage address in his local browser. Then after authorization, there are options: run all specs, run chosen specs, send report and more. It would be the best if that software could also allow simultaneous running of tests commissioned by multiple developers.
What software I need?
You need a continuous integration tool. https://stackify.com/top-continuous-integration-tools/
I recommend Jenkins.
I would first try to run your selenium tests headless in a docker container on your laptop. Once you are able to do that, use that same configuration in your docker container running in Bitbucket pipelines. It could actually be the same container and the same scripts. Then, developers can just make a branch and work with the tests on that branch. If only a certain subset of tests need to run, then the developer can make the necessary changes on his or her local branch to run those tests and push it up to Bitbucket. This should help with the configuration https://github.com/SeleniumHQ/docker-selenium.
My project is built using Vue.js, powered by Nuxt
I’m using Cypress to write/run tests
I’m using CircleCI (v2) to run the tests
Locally, using Cypress’s instance of the Electron browser on macOS, the tests run correctly.
On CircleCI, the tests display a blank screen.
I’ve SSHed into CircleCI after the tests have failed, and if I make a curl request to the local server at that address, the correct HTML is returned to me.
My hypothesis is that the browser is not able to run Vue.js, but without a console log it’s hard to tell.
Any ideas as either how to continue debugging, or even better, fix it?
Edit: I’ve written an extra test to use another site, and it loaded correctly, so I can confirm that everything else is working.
Edit: It’s in my Nuxt config, just going through it now to pull out the line causing the problem.
So, of course this was super annoying. I’m not sure what the chances are of this happening to somebody else, but here’s the answer to my problem:
Google Analytics
On my CI I don’t have a GA code, and it was falling back to empty string, so it wasn’t able to load the page. It didn’t show up on the sever logs as it was an issue on the client side, and it’s not yet possible to easily get client logs using Cypress (coming soon).
Interesting, Thomas,
Could you open a GitHub issue with a link to a small repo that we can try ourselves? I do not see any factors that might affect running Cypress tests against Nuxt app on Circle
We have e2e test integrated with Jenkins system. For a few weeks this test successfully ran both, locally and from Jenkins (as a part of the build pipeline).
At the end of Sprint, I modified the script to reflect Sprint changes and made sure it passed locally. Then, I merged the changes with master. Now, e2e runs from Jenkins are failing 100% of the time, while when I locally connect to QA envs there is no problem.
The error I am getting is - Element is not clickable at point (x, y) which I cannot reproduce locally.
The server doesn't have a real screen so I cannot go out there and see what's going on. Resolutions are perfectly matching. I have other people running this test locally and there is no problem.
What could possibly cause these failures and how do I troubleshoot this problem?
Thanks for your help!
Its a question from 1000ft and pretty difficult to identify where exactly the issue could be but I listed down some probable causes/debugging tips that could help you
1.Whats your checkout strategy from source code repository? Check job workspace and it should have the most recent code and check if its indeed the latest one.
May be configure Job to always pick a new version instead of 'update'
2.Add a reporter based on the test framework you are using especially the ones which provide screenshots. Refer my blog for more details -
3.Check the stack trace of your error from Jenkins console report and verify the exact trigerring point