I'm using TFS2015 build definitions. Basically, I have a definition with a Visual Studio Build task and then a Visual Studio Test task. In this set up, if the Build task succeeds, but the Test task fails, the entire build fails. Is there a way to get this to set the result to "partiallySucceeded?"
The reason I want to do this is for our CodedUI tests. I want to be able to test a "partiallySucceeded" build using Release Management, but currently all of our builds are either "succeeded" or "failed," even though some of the "failed" ones are only due to 1 or 2 unit tests.
Make sure you set the "continue in error" checkbox on the test task.
Related
I have set up a Build Definition to build a solution on Visual Studio Online (VSO). It's working fine, but it builds all the code every time when I check in the code.
How can I build a particular changeset from the code?
How can I use/pass this number to the "MSBuild Arguments" to use it there for deployment?
You need to turn off the CI build feature(Uncheck "Continuous integration (CI)" option under "Triggers" tab in your build definition) if you don't want it to build all the code every time when you check in.
To build a particular changeset, you can enter the changeset number in "Source Version" textbox when you queue a new build. (In Git, it is "Commit" textbox)
To pass the version number to MSBuild Arguments, you can use the pre-defined variable "$(Build.SourceVersion)", it is filled with the source version number you specified when you queue the build.
So, I have a reasonable amount of Selenium Tests. I want them to run quietly in the background via a batch script, nunit-console, and RemoteWebDriver. I have this setup already. I want to also be able to run the same tests (with me watching, debugging, writing new tests, etc...) with other drivers in visual studios 2013 using nunit. I have this already setup. The problem is I want to be able to run them at the same time.
I'm thinking of putting a check in to see if the calling program is nunit vs nunit-console to determine which driver to use, but I am a little uncertain how I should set this up.
I've considered:
bool isConsole = Process.GetProcessesByName("nunit-console")
.FirstOrDefault(p => p.MainModule.FileName.StartsWith(#"C:\Program Files (x86)\NUnit 2.6.4\bin")) != default(Process);
if (isConsole)
{
// remote
}
else
{
// ff,chrome,etc...
}
This however would not allow me to run the suite quietly in the background WHILE running individual tests in visual studios.
I'm not sure if there's any difference when you're running a selenium test, but with a normal nunit test you could do:
if("nunit" == Process.GetCurrentProcess().ProcessName)) {
...
}
This gets your the name of the process that's actually executing the tests, rather than just checking if the process is currently running on the machine.
Running from within visual studio, I get a process name of "vstest.executionengine.x86", from the console, I get "nunit-console" and from the gui I get "nunit".
It's possible that depending on the process model that you're running your tests under that you might need to check the parent process, rather than the current process. With Nunit configured to run tests in a separate process the reported process name with the above code is "nunit-agent". For some reason I can't get nunit-console to run in this mode at the moment, so I don't know if it has a different process name that you can use instead.
If you do need to trace up the process call stack to see what the parent process is, there's some excellent answers on how to do it on this question.
I am calling a custom task (derived from Microsoft.Build.Utilities.Task) from the AfterDropBuild target in my TFSBuild.proj. If my Execute override returns false, the build log shows the task as FAILED, but I still get a successful build, which means that I do not realise there is a problem with the build. How do I ensure that the build as a whole also fails?
Edit: This is TFS 2008.
You've created a mismatch between logged errors and your task result. You need to first log an error, with Log.LogError. Then return !Log.HasLoggedErrors from your tasks, always. (from trick #2 in the book "MSBuild Trickery").
Is it possible to set number of test runs variable on TFS 2010 build so that is displayed in the build summary? We are running NUnit tests and can get the number of tests run. At the moment we see "No Test Results" in build summary.
and congratulations on your promotion from shoe-salesman to TFS technician... :)
You can add the NUnit test run results to the TFS build report in one of two ways:
1. You can write a simple XSLT to transform the nunit report to a TRX file (which TFS can read).
2. You can use NUnit for Team Build.
Either way, the NUnit results will be added to the report, including the count of successful, failed, inconclusive and (IIRC) total count of tests.
Hope this helps,
Assaf.
I'm trying to integrate running Fitnesse tests from MSBuild im my nightly build on TFS.
In an attempt to make it self contained I would like to start the seleniumRC server only when it's needed from fitness.
I've seen that there is a "Command Line Fixture" but it's written in java can I use that?
I think you might be able to. You can call any process easily in MSBuild using the task. However, the problem with doing this is that the exec task will wait for the Selinium process to finish before continuing, which is not the bahaviour you want. You want to run the process, keep it running during your build and then tear it down as your build finishes.
Therefore, I think you are probably going to need to create a custom MSBuild task to do this. See the following post for an example of a tasks that someone has created that will run asynchronously returning control back to the build script:
http://blog.eleutian.com/2007/03/01/AsyncExecMsBuildTask.aspx
And for an example of calling a Java program from MSBuild (but in this case synchronously) take a look at my task that calls Ant from MSBuild here
http://teamprise.com/products/build/
As part of your MSBuild task, you will want to output the process id that you created to an output property so that at the end of your build script you can call another custom MSBuild task that kills the process. It can do this by looking for the process id passed in as a variable in MSBuild and then call Process.Kill method i.e.
Process process = Process.GetProcessById(ProcessId);
process.Kill();
That said, you would need to be careful to ensure that your kill task was always executed in MSBuild by making sure it was included during error paths etc in the build. You could probably make things a bit more resilient by making the selenium RC starter task look for other seleniumRC processes and killing them before starting a new one - that way if a process didn't get closed properly for some reason, it would only run until the next build.
Anyway - my answer sounds like a lot of work so hopefully someone else will come up with an easier way. You might be able to create the seleniumRC process in the test suite start up of the FitNesse tests and kill it in the suite tear down, or you might be able to write a custom task that extends your FitNesse runner tasks and fires up seleiniumRC asynronously before running the test process and then kills it afterwards.
Good luck,
Martin.
Thanks for your replies!
This is how I've done so far.
I made a fit fixture (very simple) that starts a process with the supplied command line, in my case startSelenium.bat. The fixture returns the ProcessID so I can store that in my fitnesse context and close that session later.
I can now make a SuiteSetUp page in my fitnesse test that looks like this.
|RunCommandFixture|
|Commandline|RunCommand?|
|C:\Projects...\startSeleniumRC.bat|>>seleniumprocess|
and a SuiteTearDown like this
|RunCommandFixture|
|ProcessID|StopCommand?|
|<
That works for me. No selenium RC starts by request from my fitnesse test.
What about writing a simple .NET app that does a Process.Start("selenumRC commandline") which gets run by your build script?
If you aren't too far down the Selenium route; might I suggest that you look at similar .NET browser automation tools; specifically WatiN or ArtOfTest. The "stacks" in these are completely .NET, so getting them running on different machines is much easier.