I'm currently trying to reduce the time taken to compile and unit test projects in TeamCity.
Currently my project takes about between 5 to 8 minutes to build.
What it does is:
Clean any existing files
Compile each project
Create the installer
Once that is done the unit tests are kicked off and it takes about 2 minutes to run.
What it does is:
Clean any existing files
Compile each project
Run all the unit tests
Now, running the unit tests only takes 5 seconds....so the clean and compile step takes about 2 minutes....and the create installer step about 3 to 6 minutes.
My first question is: Is there any way to configure team city so that it doesn't have to do the clean and compile steps again when it runs the unit tests. I believe the main reason why we have it that way is because the project build and unit tests can be run by different build agents.
My second question is: Is 5 to 8 minutes a reasonable time for a project build to take? Are there ways to optimize the compile of the projects in the solution as well as the installer creation?
Please let me know if there are any additional details I can supply that might help you to point me in the right direction of either optimising the build or just leaving things as they are.
Update to answer some questions from Nate:
When you run the build outside TeamCity, does it do a clean/compile again before the unit tests? No, because when you run it on your own machine you can specify which parts of the build script to run. We have sections for clean, compile, unit-test etc which get specified differently in team city for project build vs unit test running.
If so, can you make the build not do that before the tests? The reason we don't do this at the moment is because when different build agents are used the files needed for the unit test would not be available and thus would fail.
What are you using to build your project? we use Nant to build our project, wix for the installer, and NUnit and NCover for the unit tests and coverage reports.
What source control system are you using? Subversion
Regarding #1: When you run the build outside TeamCity, does it do a clean/compile again before the unit tests? If so, can you make the build not do that before the tests?
What are you using to build your project? What source control system are you using?
Another option consider is having multiple build configurations. At my work, we have several configurations. One that runs after each commit. It does a clean/compile and some quick tests. We then have a nightly config that does a clean/compile and all the tests. Can you do something like that?
As a rule of thumb you should keep the build that is triggered on commit as short as posible, so you should try to refactor the build configuration into a slim on commit build, and run the current configuration once every hour or so.
Related
Our goal is to implement CI testing and deployment for our DEV web environments:
Goal
Run XUnit tests on check-in.
If tests fail, create individual, associated Bug work items. Stop.
If tests tests pass, deploy build to a UNC file path.
Current Setup
CI is on for the branch, and the build definition currently has enabled Create Work Item on Failure on the Options panel.
XUnit was integrated into the Visual Studio Test build step by providing the Path to Custom Test Adapters necessary.
Problem
Tests run and display results correctly in the build, but no bugs are created for the failed tests, only one for the overall build fail.
Question
How can I create individual Bugs (and include details about the bug in its description)?
You would have to write your own code to create Bugs for each test failure.
I would however recommend against it as this creates unessesery work items and they may not really be bugs. Maybe we have a single test that fails, and the other 200 tests fail as a result. We only have one bug. You will overwhelm people.
You can easily create bugs as you investigate failures using the failed test list that is part of the build results.
https://www.visualstudio.com/en-us/docs/test/continuous-testing/getting-started/getting-started-with-continuous-testing
What I would like to do is fail and abort the TFS Build after, say 10 of the tests fail. I cannot see a parameter for this in the Build Definition.
We have 1000's of Integration tests that get run as part of a nightly build. These tests take in the region of 1 hour to run hence only the nightly build runs these; the quicker unit tests are run as part of a CI build on every check-in.
When we have a failed build at night time, we'd like to re-run that build the following day, but abort as soon as there's 10 failed tests. This will save us time when tracking down the issues.
Ideally the tests shouldn't take so long but pragmatically we cannot resolve that in a short space of time.
As far as I know, this is not possible with TFS builds out-of-the-box. The default TFS build template runs all the tests identified in the build definition in one run, without the possibility to act on the results of individual tests. The only way to accomplish this as I see it is by customizing the default TFS build template to your needs and use this template for your build definition.
You need to develop your own custom activity and use it on the build template for this purpose.
As I'm using Selenium/MSTest for UI tests, I got a problem: I need to deploy an ASP.NET site to a staging server just before the automated tests are executed during a TFS build (TFS 2012).
Although I thought I could do this configuration in the TFS Build process template (DefaultTemplate.xaml), I can't figure out how to change the order to execute a Build->Deploy->Test flow.
Note I've found some how-tos aiding in this goal when using TFS Lab Environment, but this isn't my case.
Default Template Unit Testing is different from UI Testing. Default template is designed to run Build -> Unit Test -> Deploy.
You are now in Build -> unit Test -> Deploy -> UI Test.
For UI Testing, You need separate Test Agent and set up build agent in interactive process.
These links may help
http://www.deliveron.com/blog/post/Configuring-a-TFS-2010-Team-Build-Server-to-Run-Coded-UI-Tests.aspx
http://blogs.infosupport.com/testing-your-web-application-with-selenium-and-mtm-part-1/
http://blogs.infosupport.com/testing-your-web-application-with-selenium-and-mtm-part-2/
Finally I've solved the problem chaining builds.
For example, when I queue a production build, this is the sequence:
Build, deploy to staging without UI tests
Build, no deploy, staging UI tests only
Build, deploy to production without UI tests
Build, no deploy, production UI tests only
I got the solution from this blog post:
http://blog.stangroome.com/2011/09/06/queue-another-team-build-when-one-team-build-succeeds/
Post's author posted the modified DefaultTemplate.xaml (build definition) in GitHub Gist:
https://gist.github.com/jstangroome/1196590/
Credits for the whole blogger!
One simple solution is to have an intermediate msbuild project file that is specified in your build definition and from there, call the actual solution file.
You can then do whatever is required pre- or post- build to ensure that the environment is up and running prior to TFS executing the tests. We do things like compile and deploy databases prior to TFS running our integration tests.
This is one solution that avoids having to manipulate the build template.
I looking for ways to improve build time for my maven build.
I was following this suggestion (Speed up maven) to rename the target folder to a different name and then proceed with the install goal and thus avoid the time it takes to clean.
However this leaves me with a folder that still needs to be cleaned at some stage. I was hoping that I could automate the removal of this renamed folder as a second process that runs in parallel with the install phase.
I haven't been able to find any information on whether it is possible or not.
Cheers for the help,
Ferg
I'm not aware of an out-of-the-box feature that would do that. Also the antrun plugin seems not to have an option to fork the process and run in parallel.
maybe ant itself can do that within the build script?
Maven 3 supports parallel builds: https://cwiki.apache.org/MAVEN/parallel-builds-in-maven-3.html - that might help to speed things up (but depends on the plugins you are using).
Disclaimer: I'm not at all a skilled Jenkins user.
I was looking for a plugin to allow me to run only tests affected by the latest commits, in maven-built projects, but could find neither a maven plugin smart enough to not run all tests again, nor a jenkins plugin doing this. Is there a way to achieve this?
Say, for example, I have a project consisting of A.java, testA.java, B.java and testB.java. I have jenkins configured to run a build and test for each commit. I have just commited a change to A.java, but not to B.Java, and there's no dependency from testB.java and B.java to A.java. I don't want Jenkins to run testB.java when it checks out fresh sources. If ther3e was a dependency from B.java or testB.java to A.java, I'd like jenkins to figure this out, and run testB.java too.
Background: running all tests in a project is too time consuming. Running only a fixed set of tests leaves bug holes for bugs. Running all tests which might be broken by a change by hand is error prone.