I configured xunit parser task under my project's Bamboo plan, I ran my plan:
Starting task 'Xunit' of type 'com.wwwlicious.xunit.xunit:xunitParser' Failing task since test cases were expected but none were found.
It dawns on my xunit parser only parses test result produced somehow from prev steps?
I tried to add a script task to run xunit tests manually: https://xunit.github.io/docs/getting-started/netcore/cmdline
C:\SomeApplication>dotnet test -l:trx;LogFileName=SomeApplTests.xml
Build completed.
Test run for C:\SomeApplication\bin\Debug\netcoreapp2.0\YI.LS.Test.dll(.NETCoreApp,Version=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
Total tests: 20. Passed: 20. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 4.3114 Seconds
So, okay - I can configure a Bamboo Script task to run "dotnet test", it'd produce SomeApplTests.xml under "TestResult" folder.
Now the challenge is: how to feed it to xunit parser? Also trx logger also no good, prolly need xunit logger https://github.com/spekt/xunit.testlogger
Thanks
References:
https://marketplace.atlassian.com/apps/1216522/xunit-net-parser?hosting=server&tab=support
https://github.com/wwwlicious/wwwlicious.bamboo.xunit/issues/6
How to publish results using dotnet test command
https://github.com/spekt/xunit.testlogger
Answer is found:
You need install in test project "XunitXmlTestLogger" (https://github.com/spekt/xunit.testlogger)
Then configure a Script task:
dotnet test -l:"xunit;LogFileName=Test.xml"
You need set Working Directory to point at your test project, for example: MySoln\MyTestProj
Then configure your xUnit.net Parser task in Bamboo, with xUnit Test Results File/Directory to point at: MySoln\MyTestProj*.xml (Default is: artifacts/**/*.xml)
https://github.com/wwwlicious/wwwlicious.bamboo.xunit/issues/6
Related
I am using VSTest to run tests via Azure DevOps. Tests run successfully, but using the option rerun failed tests: true, an error appears during the restart phase.
NUnit 3.12.0 ;
NUnit Adapter 3.16.1.0 (Checked with 4.0.0.0)
vstest.console.exe "C:\agent2.172.2\_work\r1\a\UITest\drop\Tests.Web\bin\Release\netcoreapp3.1\Tests.Web.dll"
/Settings:"C:\agent2.172.2\_work\_temp\3utv233tymm.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\agent2.172.2\_work\r1\a\UITest\drop\Tests.Web\bin\Release\netcoreapp3.1"
/TestCaseFilter:"FullyQualifiedName=Tests.Web.Tests._5.CourierModuleTest.N1_SendingTest.Id_5_1_01_TransferToCourierModule(Chrome)|FullyQualifiedName=Tests.Web.Tests._3.IssuanceOfDocuments.InformationOnTheApplicationIdentificationOfRecipient.N2_RecipientIdentificationTest.Id_3_2_13_RegisterAddressByFiasTest(Chrome)"
NUnit Adapter 3.16.1.0: Test execution started
An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Missing Operator '|' or '&'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
How can I fix this error and successfully restart the tests in DevOps?
I tried googling for a similar issue but didn't find anything that would work. Any help is really appreciated
Here's similar discussion, according to comments from the Contributors of azure-pipelines-task repo:
1.You should update your VS and VSTest component within it to latest version since we don't support data driven tests for Rerun scenario. It will be available with VS 15.8 release and higher.
2.You should configure your VSTest task following this solution:
Recommended to give total number of your tests as input (Number of tests per batch).
I have a TFS Release definition that runs several tests as part of the deployment:
When I check the "Rerun Failed Tests" Option in my "Visual Studio Test" step in:
My unit tests no longer run as part of my release. The release fails with the following error in the "Visual Studio Test" step:
2019-02-22T16:21:33.0983821Z ##[error]Error opening response file
'C:\TFS' 2019-02-22T16:21:33.1413767Z ##[error]The test source file
"C:\TFS Agent\ _work\r6\a\Agent_work_temp\ea14a091-36bd-11e9-99f5-51895cf3942e.txt" provided was not found.
I figured since the error started with the following statement:
Error opening response file 'C:\TFS'
That there is some sort of bug with the TFS test runner where the call to the task to execute the tests is not putting the path in quotes. I changed all my TFS build agents to using the following "type" of paths for all builds(I removed all spaces from the folder the TFS agents lived in):
C:\TFSAgent\ _work
And it seems to work now...
I have executed my tests using the command:
dotnet test -l trx
But when I saw the my test results are not appearing in the build summary:
Some hint show to make it visible? Within TFS2015, and I can't use the current build step for running tests as they do not work with dotnet core apps
This is the output of the tests in my logs:
2017-06-01T14:56:08.3158317Z Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
2017-06-01T14:56:08.3158317Z Copyright (c) Microsoft Corporation. All rights reserved.
2017-06-01T14:56:08.4228414Z Starting test execution, please wait...
2017-06-01T14:56:11.4560909Z Results File: D:\APPS\NVSDeploymentAgent\_work\12\s\tests\Tests\TestResults\2017-06-01_16_56_11.trx
To make the generated trx files available as test results in VSTS/TFS, use the "Publish Test Results" task:
I am setting up new tfs build definition using Web based build definition. I am getting an issue with the unit test cases.
If I run the build without passing any MS Build Arguments, my build is getting done successfully and it's running all test cases included in solution.
But when I pass MS Build arguments, it's not running any unit test cases - just ignoring it and going to next build step. What can be it?
I am building .NET Solution and passing MS BUILD ARGUMENTS AS
(/p:DeployOnBuild=true /P:PackageTempRootDir=\Release /p:DeployIisAppPath="Default Web Site";DefaultPackageFileName=SampleApp.zip /p:OutputPath="$(Build.BinariesDirectory)\$(BuildPlatform)\$(BuildConfiguration)")
Is there any connection between these two (MS Build arguments and Unit test) cases?
I am not getting any error, the build is also successfully passing in both cases. But unit test are not getting executed.
Here is test step output:
2017-02-14T22:35:06.6488286Z Executing the powershell script: E:\agnt\tasks\VSTest\1.0.32\VSTest.ps1
2017-02-14T22:35:07.4248374Z ##[debug]Calling Invoke-VSTest for all test assemblies
2017-02-14T22:35:07.6348369Z Working folder: E:\agnt\_work\39
2017-02-14T22:35:07.6348369Z Executing C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "E:\agnt\_work\39\s\CaseAPI\Main\packages\Agero.TestRunner.1.0.0\lib\net45\Agero.TestsRunner.NUnit.dll" /TestCaseFilter:"TestCategory!=Ignore" /EnableCodeCoverage /logger:trx
2017-02-14T22:35:07.8428336Z Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1
2017-02-14T22:35:07.8438332Z Copyright (c) Microsoft Corporation. All rights reserved.
2017-02-14T22:35:07.9678332Z Starting test execution, please wait...
2017-02-14T22:35:08.0108349Z Warning: Using Isolation mode to run the tests as diagnostic data adapters were enabled in the runsettings. Use the /inIsolation parameter to suppress this warning.
2017-02-14T22:35:16.0528901Z Attachments:
2017-02-14T22:35:16.0538895Z E:\agnt\_work\39\TestResults\6eaffd7e-b2ac-4bc4-b356-41a2fda7bad9\csr1_B-TFSBC001WV 2017-02-14 17_35_10.coverage
2017-02-14T22:35:16.2028915Z Information: Additionally, you can try specifying '/UseVsixExtensions' command if the test discoverer & executor is installed on the machine as vsix extensions and your installation supports vsix extensions. Example: vstest.console.exe myTests.dll /UseVsixExtensions:true
2017-02-14T22:35:16.3698948Z ##[warning]No results found to publish.
Any advice/help is appreciated!
As you have changed the OutputPath to $(Build.BinariesDirectory)\$(BuildPlatform)\$(BuildConfiguration), so you also need to specify the test assembly in VS test task to: $(Build.BinariesDirectory)\$(BuildPlatform)\$(BuildConfiguration)\*test*.dll:
I'm successfully generating 2 .exec files by Jacoco within "build/jacoco" folder after running a Gradle based build and integration tests.
Gradle command:
"gradle clean build integrationTest"
Once done, it generates the following .exec files under build/jacoco folder.
test.exec
integrationTest.exec
Following is my sonar-project.properties file. When, I run "sonar-runner" from Linux prompt it completes but on SonarQube dashboard for this project, I see Unit test says some 34.5% but integration tests says 0.0%. Both .exec files have valid size. I also did "cat" on the .exec files and piped the output to "strings" command in Linux and saw that integrationTest.exec did hit the Tests functions - I have only 1 .java file.
When I run "gradle clean build integrationTest sonarRunner -Dxxx.xxx=yyy -Dyyy.xx=zzz" i.e. by passing all the sonar variable as mentioned in the sonar-project.properties file using -D option, it works but same result on SonarQube project's dashboard. Project's sonar dashboard has both widgets configured for Unit / Integration Tests and I'm including IT tests for showing Overall coverage. Overall coverage is showing 34.5% (which is Unit test % value). Sonar does see test.exec, integrationTest.exec and also auto generates overall-xxx.exec file as well during this operation.
NOTE: I'm no where - while starting tomcat on a separate putty / linux console -OR within Gradle build script, providing any value or setting JAVA Agent for Jacoco. I'm getting integrationTest.exec file and test.exec file already so not sure if JVM needs to be stopped once IT tests are complete running. I don't think I need those as i have valid file size for .exec files.
My ?:
- Why sonar is not getting IT coverage on the dashboard even though I'm setting / passing the following variable correctly:
sonar.jacoco.itReportPath=build/jacoco/integrationTest.exec
-bash-3.2$ cat sonar-project.properties
# Root project information
sonar.projectKey=com:company:product:ProjectA
sonar.projectName=ProjectA
sonar.projectVersion=1.0
# optional description
sonar.projectDescription=ProjectA Service
#Tells SonarQube that the code coverage tool by unit tests is JaCoCo
sonar.java.coveragePlugin=jacoco
#Tells SonarQube to reuse existing reports for unit tests execution and coverage reports
sonar.dynamicAnalysis=reuseReports
# Some properties that will be inherited by the modules
sonar.sources=src/java,test/java,src/java-test
# Sonar Unit Test Report path
sonar.jacoco.reportPath=build/jacoco/test.exec
# Sonar Integration Test Report Path
sonar.jacoco.itReportPath=build/jacoco/integrationTest.exec
sonar.junit.reportsPath=build/UT/results
# Sonar Binaries
sonar.binaries=build/classes/main
Narrowing down the cause: I think it's due to the .exec file for Integration test. To proove it: I passed UT exex file to both reportsPaths in Sonar variables i.e. the following and SonarQube picked both UT/IT test coverage. This prooves that if .exec file for IT tests is good (which I think it's But I need to double check) then Sonar will pick the .exec file and show a valid coverage % instead of 0.0%. Note: the following is just to proove if Sonar is picking the values or not. itReportPath variable should use the .exe file which is generated by Integration tests by Jacoco.
sonar.jacoco.reportPath=build/jacoco/test.exec
# Sonar Integration Test Report Path
#sonar.jacoco.itReportPath=build/jacoco/testintegrationTest.exec
sonar.jacoco.itReportPath=build/jacoco/test.exec
OK Found the issue. I was running integrationTest task in Gradle and was NOT attaching the jacocoagent.jar (as per Jacoco documentation) to the target JVM (Tomcat's instance) scope. Once I did that, I removed jacoco { ... } section from integrationTest task in Gradle (build.gradle or GRADLE_HOME/init.d/some.common.gradle file as this attach jacoco agent to the Java JVM in which Gradle is running). Now, once jacocoagent.jar was attached to Tomcat's JVM (as per the line below which I added in Tomcat's startup.sh script and added the variable to the command which starts Tomcat), then I ran Gradle (integrationTest) task for running IT tests.
PROJ_EXTRA_JVM_OPTS=-javaagent:tomcat/jacocoagent.jar=destfile=build/jacoco/IT/jacocoIT.exec,append=false
Then while Gradle was in progress, tests ran and I got a file (jacocoIT.exec at the given location) with some file size BUT this is not yet the final one. I had to stop the Tomcat session/JVM instance by running Tomcat's stop.sh script. Once Tomcat was stopped, I saw jacocoIT.exec file size increased significantly and this was the valid final jacocoIT.exec file (which I needed for sonarRunner Gradle task OR sonar-runner exectuable to pick and successfully push IT code coverage data to project's sonar dashboard). Once done, I got both UT + IT and it's combined code coverage.
sonar.jacoco.reportPath=build/jacoco/UT/jacocoUT.exec
sonar.jacoco.itReportPath=build/jacoco/IT/jacocoIT.exec