What is the best practice for running integration/e2e tests after build with CI? Seems like the common flow is build, test, deploy, however running integration test/e2e requires a build/staging server to run something like selenium. Does anyone have a hello world example of this or example .gitlab-ci.yml file?
Related
I have to merge my test automation code base with my application development code base in same repo, i have 2 different pom's- one for development code and other for automation testing code.
Dependencies used in development code are not there in testing code and vice-versa. Currently i have merged both the pom's and able to run automated tests on CI/CD pipeline.
But need to have separate pom's for both of them as it might take more time for dev code to build/deploy on CI/CD pipeline because of dependencies present in testing pom.
Please suggest the efficient way to do this?
I'm trying to improve the testing process where I work, but without adjusting the structure.
What we have: VSTS, Selenium IDE, Testers who write test cases, but not code.
What I'd like to do is manage a way to marry our TFS continuous integration with the Selenium tests we write. These are NOT the code-driven selenium tests, but rather the IDE version where users click through, and set assertions using the IDE (All are just UI tests). I know we can export those tests plans as a .SIDE file, but what I can't figure out, is how to have our TFS server execute those as part of a deployment or build pipeline.
Ideally, developers/devops would setup projects in TFS from the onset with whatever solution makes sense to execute these Selenium .SIDE files, but afterwards, the testers would manage adding/modifying those tests cases elsewhere.
The real goal here is to not have testers writing code, or checking in code. Only writing these UI Selenium tests, but having TFS execute those as part of CI.
Researching this on the internet drives me basically always to something that requires testers to write code.
I don't think it can automate testing without code, at lease, you need a test project containing your automated tests.
Generally, in Azure DevOps, we use Visual Studio Test task to run tests. This task supports using the following tests:
Test assembly: Use this option to specify one or more test assemblies that contain your tests. You can optionally specify a
filter criteria to select only specific tests.
Test plan: Use this option to run tests from your test plan that have an automated test method associated with it. To learn more about
how to associate tests with a test case work item, see Associate
automated tests with test cases.
Test run: Use this option when you are setting up an environment to run tests from test plans. This option should not be used when
running tests in a continuous integration/continuous deployment
(CI/CD) pipeline.
This was a question that I had as well, and I think I found an imperfect but better solution.
I wasn't able to get my Selenium IDE tests running with Jenkins, but I was able to get them to run with TeamCity, another CI.
I created a build step like the following :
Runner type: Command Line
Working Directory: where the selenium IDE .side file is located
Run: Custom Script
With the build script content that I usually use to run my Selenium IDE Tests, such as selenium-side-runner sidefile.side
I also added the following so I could output the results in Junitor another form: --output-directory=results --output-format=junit
You can also add the following so the tests are run headlessly, this only works in Chrome : -c "goog:chromeOptions.args=[--headless,--nogpu] browserName=chrome"
Finally, I also use --filter to run one test suite at a time, but that is optional too.
I then used another build step to export the results to our test manger, xray, but I think that is beyond the scope of this question.
The problem with this solution is that it runs directly from a users individual machine still, but this can be work around.
I have created a Jenkins maven task to run selenium tests on one project, and now I want to use these selenium test in a proper way for CI.
Actually, I have a Jenkins task which 1-builds the project, 2- uses sonar, 3 - deploys the project. I would like to add the selenium tests to this process. The question is: can I run the selenium tests before deployment? Is it necessary to do a previous deploy for the selenium tests before the real deployment? Is there anyway to simulate a deployment or something like that so I can run the selenium tests?I would appreciate If you could advise me on how to do or any plugin which could help me.
As per my understanding, To deal with the scenario you should run the selenium test on QA environment. If all test pass then deployment should start for staging etc.
Additionally, once the deployment is done on staging, then selenium script should run again, test staging and if something went wrong then rollback from staging should be happened.
I never tried it but you can use below github plug-in for deployment(in case you are using github) :-
https://wiki.jenkins.io/display/JENKINS/GitHub+plugin
If you are using SVN then use below :-
https://www.packtpub.com/mapt/book/application_development/9781783553471/3
Hope it will help you :)
I'm building a project where we have to run end-to-end tests with Selenium like so: Run focused integration or End-to-end tests (e.g. Selenium). It is necessary to run this on an external staging server (e.g. Heroku). To run the integration test the application needs to connect to an external system e.g. database.
This very likely has something to do with our .travis.yml file, which looks something like this now (even though we have gone very back-and-forth with the file):
...
script:
- ./gradlew check
deploy:
provider: heroku
api_key:
secure: *****
app: *****
after_deploy:
- ./gradlew seleniumXvfb
Basically, what we want to do is first run ./gradlew check which runs unit tests, then deploy the application to heroku and finally run Selenium tests (end-to-end tests) on the staging server (heroku).
But, what happens is that travis doesn't seem to care that the selenium tests fails when it should fail. Travis shows the green checkmark for the build in whole, like everything is OK.
When this is all over, we want to deploy to a production server.
Thank you.
after_deploy currently doesn't fail the build in Travis CI.
If you want to test your application against a running staging system on Heroku, then I'd recommend deploying this manually as part of the before_script step and then running the ./gradlew seleniumXvfb command in your script section.
That way you can then do a proper production deployment based on the success of testing against your staging system.
I have setup Bamboo to run JBehave tests on a remote agent (with JBehave-web plugin launching test using webdriver), and everything runs fine. Only problem is after the execution is finished Bamboo shows no test executed. I can see the option in Bamboo to select the output of the test results, but it has to be a JUnit xml, and Jbehave reports are only generated in plain text or html.
Any idea how to solve this?
Thanks
I ran in the same situation about a year ago. JBehave "doesn't" integrate with Bamboo out of the box. Although, they have a plugin for Hudson CI.
In my case, such as yours, I resorted in running the tests through the Surefire plugin; the outputs are considered as JUnit tests results and Bamboo can recognize them.
Hope it helps.
There is a really simple way to do this. And I'm currently doing this for our build system.
Write a simple parse script that transforms your plain of html report into JUnit compatible results. And add that script as a task in your Bamboo task, then use Junit parser to parse the results. Boo! You're done! Plus, you've got the capability to quarantine!
This is way much faster than writing a plugin for Bamboo, which involves considerable more time to learn/write.
Setup JBehave with Maven.In Bamboo build plan use Maven task to run it. For getting results in Bamboo use JBehave Task for Bamboo. It will convert JBehave scenarios in tests in Bamboo. If scenario names contains JIRA issue ids, it will link them to JIRA issues.
https://marketplace.atlassian.com/plugins/com.mdb.plugins.jebehaveforbamboo/server/overview
Sample JBehave as Maven Project
https://bitbucket.org/vikasborse/jbehavesampleproject/overview
Download or clone this repository on your local machine.
To run navigate to this project in command line and use command:
"mvn integration-test"