We are building apps for iOS and Android. Currently we are experimenting with TeamCity for CI.
Currently we have various tools to help us in the Test automation.
1. Selenium: For testing the web interface
2. SOAPUI: For testing the web services
3. Calabash-iOS: For testing iOS app
This is the scenario that we have done successfully with TeamCity:
1. Developer commits code to BitBucket
2. TeamCity runs calabash
3. If calabash pass, TeamCity will build and give the artifacts
This is the scenario that I am trying to do so that I can achieve an end-to-end automated testing:
When developer commits his code to our Bitbucket git repository,
TeamCity will trigger Calabash to run and check for the UI and the output of the details. On end,
TeamCity will trigger Selenium to add data in the Back Office web. On end,
TeamCity will trigger SOAP UI to check newly added data. On end,
TeamCity will trigger Calabash to check the newly added data
Has anyone tried this before or can give me an idea on how I can approach this?
You can setup different builds for each step since you can trigger builds via simple GET requests triggered by a script. TeamCity Accessing Server by HTTP
We run a similar setup. Basically we have one build kick off the Selenium/SoapUI scripts. That build doesn't wait for the job to finish, it just kicks it off. Then in our Selenium/SoapUI tests as the last step we hit the URL for the next build that picks up reports etc.
Related
I am working using TFS to create test case. I have a automated test written in Robotframework with Selenium2Library.
I'd like to be able to associate these tests with test cas in test. Is it possible to link the automated test cases with the test case in TFS, and if so, how is it done?
The web Test page does not support to run automatic tests associated with test cases in TFS.
You can achieve the same goals using the Build & Release features in Team Foundation Server (TFS). (https://www.visualstudio.com/en-us/docs/test/lab-management/use-build-or-rm-instead-of-lab-management). Simply integrate Selenium testing into a continuous integration/continuous delivery pipeline.
Instead of build, suggest you use a step in a release pipeline. Take a look at this question:How can i run automated selenium tests using TFS build 2015?
More detail steps about how to integrate with TFS please go through the official tutorials from MSDN: Get started with Selenium testing in a CD pipeline
Currently, I am doing automation UI tests with testcafejs for React project.
I would like to run these tests in continuous integration environments such as Jenkins. I already add browserStack plugin to run them locally with different browsers and operating system.
However, I would like to integrate them into Jenkins but not always consuming browserStack quota for daily checks. Any suggestions for how to run them within different browsers in Jenkins?
TestCafe has an extensive command line interface that allows it to fit well in any popular continuous integration system.
Here are instructions on how you can integrate TestCafe with Jenkins. The up-to-date KB article can be found in this documentation topic for the TestCafe Open Source version:
Integrating TestCafe with Jenkins CI System
If you are using the legacy TestCafe version (version 15.1), you can use the following KB article:
How to integrate TestCafe with Jenkins
You may also wish to check the TestCafe plugin for Jenkins that attaches screenshots and videos to the Jenkins test results page.
Step 1 - Fetching Test Code From a Repository
Here, we will use tests published in a separate repository on GitHub - ci-integration-demo. If you use a different version control system, search for a plugin that integrates it with Jenkins.
Open your project and choose Configure from the right pane.
Scroll down to the Source Code Management section and select Git, then specify the Repository URL.
Step 2 - Adding a Command to Install TestCafe
Go to the Build section, find a step that builds you application and add a new step right after it. To do this, click Add build step and select a step type that runs a shell command.
In the Command box, type the following.
npm install testcafe testcafe-reporter-xunit
This command installs the main testcafe module and a plugin that saves test run reports in the xUnit format.
Step 3 - Adding a Command to Run TestCafe
Add another step that executes a shell command after the previous one. This step will run TestCafe.
Type the following command.
node_modules/.bin/testcafe chrome tests/**/* -r xunit:res.xml
This runs TestCafe tests from the tests directory in Google Chrome. Test results are saved to the res.xml file in the xUnit format.
Step 4 - Publishing Test Run Reports
Go to the Post-build Actions section and click Add post-build action. In the drop-down list, select Publish JUnit test result report.
In the Test report XMLs field, specify the test report file: res.xml.
Step 5 - Run the Test
Click Save and you will be navigated to the Project page.
Hit Build Now to build the project immediately.
Step 6 - View Test Results
In the Build History section of the Project page, click a build and select Test Results from the drop-down menu.
Jenkins will display a test run report where you can see general information about testing results. You can click individual tests for details.
I am evaluating AWS device farm for running mobile web test. These are the steps I am anticipating:
Create sample tests (Java)
Package that as zip file
Go through the AWS device farm console and upload the test zip only
Manually select the configuration and other things
Manually execute the test and evaluate results
For thing I need help are:
a) What if the tests needs some changes, do I need to go through the JAR package creation for every run? Can I run test from my IDE and if everything works fine then only package and upload it on AWS device farm?
b) To do (a) I noticed they have API to simulate step 3-5 to achieve running tests but wondering if there is some easy way to do it?
The steps you've listed are the correct sequence of tasks that need to be performed to run tests on a device. With AWS Device Farm, you have to perform the extra step of uploading the tests and application to the service. As you stated, every time you change your tests, you will need to rebuild the JAR and upload it. Most customers set up a continuous build/integration pipeline using a tool like Jenkins to perform this task automatically.
If you are running in Android Studio, you can use the Device Farm Gradle plugin, which will do the work for you.
I'm new to Jenkins, so please go easy!
I have a web application which we are developing, and we've started automating our release using Jenkins.
I also have a standalone Selenium WebDriver script which will perform a Smoke test on our web app. We usually run this manually each time there's a new deployment.
I heard Jenkins can automatically trigger Selenium tests. So what I did was to create a batch file, which will in turn call the Selenium script. I added a Build Step which will call this batch file.
What's happening now is Jenkins first Builds the WAR file, executes the batch (for selenium) and then deploys it to the target Tomcat.
But I was wondering if I could change the order to Build WAR --> Deploy to Tomcat --> Call the Batch file that executes Selenium Test. I want to do this as Jenkins Tests before deploying, which means my Selenium test still runs on the old build. This makes little sense, as I would rather run the Selenium test on the new build.
In short, is there a way I can execute the Batch file as part of a Post Build Step rather than a Build Step?
Thank you Würgspaß !!
I solved my problem by creating a separate Selenium Job which gets triggered automatically if my Build is successful. This way, I can create any number of downstream jobs to be triggered for a successful build.
I was wondering if it is possible for Jenkins to be notified after TFS finishes a build.
I'm trying to make a system where, after the TFS build, Jenkins is used to run automated selenium tests.
Thanks in advance!
You have a few options:
You could use a GenericTest to trigger Jenkins and include as part of the Build Definition. This also allows your script to communicate pass/fail information back to the build.
You could just include an InvokeProcess workflow activity to trigger Jenkins. The process would be similar to this or this.
Or you could use a tool that would trigger after a Team Build has finished. Normally people would use tools that then deploy stuff. Most of these tools trigger a Powershell script at some point, which would allow you to trigger the test run:
Octopus Deploy
TFS Deployer