I am creating a TeamCity build configuration with a few steps. One of those steps is running tests using MSTest. Currently, my tests are failing (by design so I can test the build process), but the build steps after the step for running the tests happen, even though I can see the test process exits with code 1 (ie, something other than code 0) in the build log. It does mark the build as failed, but I'd prefer it if the steps in the build stopped once failing tests were detected. Is that possible and I'm just missing something in the configuration?
When you create build step there are select Execute step for execution policy. Where you should select Only if all previous steps were successful.
Here is it:
But in your case there are bug in the MsTest and NUnit build steps. Here is workaround. And here is related issue.
Related
I am using VSTest to run tests via Azure DevOps. Tests run successfully, but using the option rerun failed tests: true, an error appears during the restart phase.
NUnit 3.12.0 ;
NUnit Adapter 3.16.1.0 (Checked with 4.0.0.0)
vstest.console.exe "C:\agent2.172.2\_work\r1\a\UITest\drop\Tests.Web\bin\Release\netcoreapp3.1\Tests.Web.dll"
/Settings:"C:\agent2.172.2\_work\_temp\3utv233tymm.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\agent2.172.2\_work\r1\a\UITest\drop\Tests.Web\bin\Release\netcoreapp3.1"
/TestCaseFilter:"FullyQualifiedName=Tests.Web.Tests._5.CourierModuleTest.N1_SendingTest.Id_5_1_01_TransferToCourierModule(Chrome)|FullyQualifiedName=Tests.Web.Tests._3.IssuanceOfDocuments.InformationOnTheApplicationIdentificationOfRecipient.N2_RecipientIdentificationTest.Id_3_2_13_RegisterAddressByFiasTest(Chrome)"
NUnit Adapter 3.16.1.0: Test execution started
An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Missing Operator '|' or '&'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
How can I fix this error and successfully restart the tests in DevOps?
I tried googling for a similar issue but didn't find anything that would work. Any help is really appreciated
Here's similar discussion, according to comments from the Contributors of azure-pipelines-task repo:
1.You should update your VS and VSTest component within it to latest version since we don't support data driven tests for Rerun scenario. It will be available with VS 15.8 release and higher.
2.You should configure your VSTest task following this solution:
Recommended to give total number of your tests as input (Number of tests per batch).
I've got some tests setup to run via Taurus and kicking them off in a Jenkins stage like so:
...previous stages...
stage('Load Tests'){
dir('./tests'){
bat "bat _testFile.yml"
}
}
...stages to execute if Load Tests stage succeeds
I want to bail out of the whole build if any one of the iterations in any of my tests fails. But this setup, as well as wrapping in a try/catch don't work.
My fault. The fail criteria were not properly implemented. Now that they are, the above 'successfully' fails the jenkins build when the criteria are not met.
Using gradle 2.2.1 and com.android.tools.build:gradle:1.5.0
When I run ./gradle build in the console the build fails at step :testDebugUnitTest.
It fails because it tries to run a InstrumentationTestCase that should not happen.
What am I missing here?
This is normal. It's because build depends on other tasks. You can see these tasks by running gradle tasks --all.
For build it will show this: build - Assembles and tests this project. [assemble, check].
The check task is configured like this: check - Runs all checks. [assembleDebug, lint, test]
The test task like this: test - Run unit tests for all variants. [testDebugUnitTest, testReleaseUnitTest]
And that is why you see the testDebugUnitTest show up when you run the build task.
So if you just want to build without testing, you need to run gradle assemble. And if you want to save time and only build the variant you need, you can run gradle assembleDebug, for example.
I'm running TeamCity 6.5 on a Windows Server, with a couple of build agents on the same server (all running as the system user as services). I had been building SilverLight projects and running the StatLight (v 1.4.4147) tests previously under Jenkins with no problems. On Jenkins, I called the StatLight test in a custom script as follows:
StatLight.exe -x="Tests.xap"
StatLight.exe -x="MoreTests.xap"
StatLight.exe -x="EvenMoreTests.xap"
... etc., but when I migrated my build jobs to TeamCity, I also changed these into a single command line step as follows:
StatLight.exe --teamcity -x="Tests.xap" -x="MoreTests.xap" -x="EvenMoreTests.xap"
This works about 50% of the time, but when it fails, there's no output in the build log to tell me why - I just get:
[11:41:18]: [MyProject\bin\Release\MoreTests.xap] Tests.ExtensionsTests.WatchObservableCollection
[11:41:18]: [MyProject\bin\Release\MoreTests.xap] Tests.SubscribingModelBaseTests.DisposeIsCalled
[11:41:18]: [MyProject\bin\Release\MoreTests.xap] --- Completed Test Run at: 28/09/2011 11:41:18. Total Run Time: 00:00:11.8125000
[11:41:19]: [MyProject\bin\Release\MoreTests.xap] Test run results: Total 6, Successful 6, Failed 0,
[11:41:19]: [Step 5/6] MyProject\bin\Release\EvenMoreTests.xap (9m:42s)
... and then nothing more. The time reported in that last line just goes up and up until I kill the the build job. Adding the --debug switch to StatLight doesn't improve the above output either.
Right now, I've switched the TeamCity build step to call each test individually as I was in Jenkins, but this is more of a workaround than a proper solution. And of course, I may still run into the above problem - I've yet to find out.
What I'd like to know is what steps I can take to debug this issue properly, or whether there are known issues that can cause the above behaviour?
There was one issue fixed in the 1.5 version relating to teamcity. http://statlight.codeplex.com/workitem/13654
I'm not sure it will fix your issue, but would you mind upgrading, trying and reporting back?
nunit tests fails when run through cc.net saying process timeout. Process has been killed
All works fine when through nUNit or VS.
Also cc.net will then show the results of previous build even if the build is a clean one.
Any help plz.
The default timeout is 600 seconds. If your tests start to exceed that the build will fail with no indication. You may need to up the timeouts for your cc.net nunit task
If you are seeing the results from a previous build, it is probably because you are not deleting the results from your previous build.
For example, my NUnit test results are written to files with the name {foo}-results.xml:
<publishers>
<merge>
<files>
<file>bin\debug\*-results.xml</file>
</files>
</merge>
</publishers>
In my tasks, I have a step in my build file that deletes the entire "bin\debug" directory so that my results are always the current ones.
One possibility is that you have a permission issue. CruiseControl is perhaps running under a service account and has different permissions than your user account (which I'm assuming you use to manually run the tests.) Try logging into the machine as the service account, then see if you can run the unit tests through VS or NUnit.
I've seen this happen if a test has an assertion, e.g. Debug.Assert(something here). When this happens to me in CC.Net, the CC.Net build pops up a message box for the assertion. Since no one closes out the message box on the build server, the NUnit test times out.