We use Hudson for our build/CI needs. In addition to unit tests (running during build) I have a staging environment that runs additional integration tests. Basically the build happens and then build artifacts are submitted the external system. I do not wish to block a Hudson build to wait for integration tests (as that locks the station into idle, and prevents it from building anything else). What I want is to update the the result of the build with the result of the external tests (and attach some logs back to the build, if possible).
Now because the staging environment is asynchronous to the build system (i.e. other systems/people can submit tests), Hudson can't be just monitoring what goes on there right after the build. Hudson build just goes into a test queue. So, I need to notify Hudson, it can't be polling something for updates.
Does Hudson support such behaviour, and if so, how can I achieve it?
I would suggest using the hudson users mailing list [1]
[1] http://java.net/projects/hudson/lists/users/archive
To solve the asynchronous wait issue, you can use the build triggering with an authentication token, and use it in a script.
The Build Triggers section has a Trigger Builds Remotely (e.g. from scripts), which when selected allows you to enter an Authentication Token (which has the descriptive text from your question).
You can thus, remotely trigger the build from a script, i.e., have it as part of the integration test script and trigger the build job using this authentication token.
..
You can have a downstream project just for result collection that integrates the results from various tests and report it to the master and aggregate them all. This project can be triggered using the authentication token, or if there is a single integration test job, you can tie it up as a downstream project.
Related
We are working with a client that uses VeriFIX to test their FIX message flow. Whilst they have built up lots of tests in many suites, it is a manual process to run them and to collate the results.
On the VeriFIX website it says
Incorporate tests into nightly builds using VeriFIX’s command-line script player.
but I cannot find any details on how to to it. Does anyone have any experience in running VeriFIX tests in a continuous integration server (ideally a Jenkins pipeline).
Many thanks.
You can run VeriFIX playlists in batch mode from the command line:
"%VERIFIX_HOME%\verifixbatch\verifixbatch.exe" -version "FIX (x.y)" -playlist "myplaylist" -disablelogging "false"
If you have received the user manual with your installation of veriFIX, the details of how to integrate with CI are in there.
To integrate veriFIX with Jenkins you will create batch files containing tests and run the batch files as jobs in Jenkins.
The placement of your veriFIX installation is important. If your veriFIX is on a users machine, as is often the case, separate from the environment machine Jenkins resides on, there can be difficulties getting the tests to run.
If you have a centralised install of veriFIX things are much easier.
Our goal is to implement CI testing and deployment for our DEV web environments:
Goal
Run XUnit tests on check-in.
If tests fail, create individual, associated Bug work items. Stop.
If tests tests pass, deploy build to a UNC file path.
Current Setup
CI is on for the branch, and the build definition currently has enabled Create Work Item on Failure on the Options panel.
XUnit was integrated into the Visual Studio Test build step by providing the Path to Custom Test Adapters necessary.
Problem
Tests run and display results correctly in the build, but no bugs are created for the failed tests, only one for the overall build fail.
Question
How can I create individual Bugs (and include details about the bug in its description)?
You would have to write your own code to create Bugs for each test failure.
I would however recommend against it as this creates unessesery work items and they may not really be bugs. Maybe we have a single test that fails, and the other 200 tests fail as a result. We only have one bug. You will overwhelm people.
You can easily create bugs as you investigate failures using the failed test list that is part of the build results.
https://www.visualstudio.com/en-us/docs/test/continuous-testing/getting-started/getting-started-with-continuous-testing
We are building apps for iOS and Android. Currently we are experimenting with TeamCity for CI.
Currently we have various tools to help us in the Test automation.
1. Selenium: For testing the web interface
2. SOAPUI: For testing the web services
3. Calabash-iOS: For testing iOS app
This is the scenario that we have done successfully with TeamCity:
1. Developer commits code to BitBucket
2. TeamCity runs calabash
3. If calabash pass, TeamCity will build and give the artifacts
This is the scenario that I am trying to do so that I can achieve an end-to-end automated testing:
When developer commits his code to our Bitbucket git repository,
TeamCity will trigger Calabash to run and check for the UI and the output of the details. On end,
TeamCity will trigger Selenium to add data in the Back Office web. On end,
TeamCity will trigger SOAP UI to check newly added data. On end,
TeamCity will trigger Calabash to check the newly added data
Has anyone tried this before or can give me an idea on how I can approach this?
You can setup different builds for each step since you can trigger builds via simple GET requests triggered by a script. TeamCity Accessing Server by HTTP
We run a similar setup. Basically we have one build kick off the Selenium/SoapUI scripts. That build doesn't wait for the job to finish, it just kicks it off. Then in our Selenium/SoapUI tests as the last step we hit the URL for the next build that picks up reports etc.
As I'm using Selenium/MSTest for UI tests, I got a problem: I need to deploy an ASP.NET site to a staging server just before the automated tests are executed during a TFS build (TFS 2012).
Although I thought I could do this configuration in the TFS Build process template (DefaultTemplate.xaml), I can't figure out how to change the order to execute a Build->Deploy->Test flow.
Note I've found some how-tos aiding in this goal when using TFS Lab Environment, but this isn't my case.
Default Template Unit Testing is different from UI Testing. Default template is designed to run Build -> Unit Test -> Deploy.
You are now in Build -> unit Test -> Deploy -> UI Test.
For UI Testing, You need separate Test Agent and set up build agent in interactive process.
These links may help
http://www.deliveron.com/blog/post/Configuring-a-TFS-2010-Team-Build-Server-to-Run-Coded-UI-Tests.aspx
http://blogs.infosupport.com/testing-your-web-application-with-selenium-and-mtm-part-1/
http://blogs.infosupport.com/testing-your-web-application-with-selenium-and-mtm-part-2/
Finally I've solved the problem chaining builds.
For example, when I queue a production build, this is the sequence:
Build, deploy to staging without UI tests
Build, no deploy, staging UI tests only
Build, deploy to production without UI tests
Build, no deploy, production UI tests only
I got the solution from this blog post:
http://blog.stangroome.com/2011/09/06/queue-another-team-build-when-one-team-build-succeeds/
Post's author posted the modified DefaultTemplate.xaml (build definition) in GitHub Gist:
https://gist.github.com/jstangroome/1196590/
Credits for the whole blogger!
One simple solution is to have an intermediate msbuild project file that is specified in your build definition and from there, call the actual solution file.
You can then do whatever is required pre- or post- build to ensure that the environment is up and running prior to TFS executing the tests. We do things like compile and deploy databases prior to TFS running our integration tests.
This is one solution that avoids having to manipulate the build template.
I was wondering if it is possible for Jenkins to be notified after TFS finishes a build.
I'm trying to make a system where, after the TFS build, Jenkins is used to run automated selenium tests.
Thanks in advance!
You have a few options:
You could use a GenericTest to trigger Jenkins and include as part of the Build Definition. This also allows your script to communicate pass/fail information back to the build.
You could just include an InvokeProcess workflow activity to trigger Jenkins. The process would be similar to this or this.
Or you could use a tool that would trigger after a Team Build has finished. Normally people would use tools that then deploy stuff. Most of these tools trigger a Powershell script at some point, which would allow you to trigger the test run:
Octopus Deploy
TFS Deployer