I have problem in Bamboo 4.4.4:
I've created plan which start some tests and then it generate TestReport.xml.TestReport
I try to parse it with JUnit Parser, but in Test Results Bamboo show only successfull tests and not even one failed.
Any suggestions?
P.S. I atached generated my TestReport.xml and xml-file, that created Bamboo after parse.buildresults19.txt
If your having issues with getting the tests to report correctly please contact Atlassian Support - they can help you sort it out quicker than Stack Overflow!
Thanks
James Dumay
Bamboo Product Manager
Related
I am using VSTest to run tests via Azure DevOps. Tests run successfully, but using the option rerun failed tests: true, an error appears during the restart phase.
NUnit 3.12.0 ;
NUnit Adapter 3.16.1.0 (Checked with 4.0.0.0)
vstest.console.exe "C:\agent2.172.2\_work\r1\a\UITest\drop\Tests.Web\bin\Release\netcoreapp3.1\Tests.Web.dll"
/Settings:"C:\agent2.172.2\_work\_temp\3utv233tymm.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\agent2.172.2\_work\r1\a\UITest\drop\Tests.Web\bin\Release\netcoreapp3.1"
/TestCaseFilter:"FullyQualifiedName=Tests.Web.Tests._5.CourierModuleTest.N1_SendingTest.Id_5_1_01_TransferToCourierModule(Chrome)|FullyQualifiedName=Tests.Web.Tests._3.IssuanceOfDocuments.InformationOnTheApplicationIdentificationOfRecipient.N2_RecipientIdentificationTest.Id_3_2_13_RegisterAddressByFiasTest(Chrome)"
NUnit Adapter 3.16.1.0: Test execution started
An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Missing Operator '|' or '&'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
How can I fix this error and successfully restart the tests in DevOps?
I tried googling for a similar issue but didn't find anything that would work. Any help is really appreciated
Here's similar discussion, according to comments from the Contributors of azure-pipelines-task repo:
1.You should update your VS and VSTest component within it to latest version since we don't support data driven tests for Rerun scenario. It will be available with VS 15.8 release and higher.
2.You should configure your VSTest task following this solution:
Recommended to give total number of your tests as input (Number of tests per batch).
As I am implementing an automated way to GUI test our webapplication with selenium I ran into some issues.
I am using selenese-runner to execute our Selenium test suites, created with Selenium IDE as a post build action in Jenkins.
This works perfeclty fine, as the build fails when something is wrong, and the build succeeds if all tests are passed. And the results are stored on a per build basis as HTML files, generated be selenese-runner.
My problem is however, that I seem to be unable to find a way, how to display these results in the respective jenkins build.
Does anyone have an idea how to solve this issue. Or maybe I am on the wrong path at all?
Your help is highly appreciated!
I believe the JUnit plugin should do what you want, but it doesn't work for me.
My config uses this shell script to run the tests (you can see the names of all my test suites):
/usr/bin/Xvfb &
export DISPLAY=localhost:0.0
cd ${WORKSPACE}
java -jar ./test/selenium/bin/selenese-runner.jar --baseurl http://${testenvironment} --screenshot-on-fail ./seleniumResults/ --html-result ./seleniumResults/ ./test/selenium/Search_TestSuite.html ./test/selenium/Admin_RegisteredUser_Suite.html ./test/selenium/Admin_InternalUser_Suite.html ./test/selenium/PortfolioAgency_Suite.html ./test/selenium/FOAdmin_Suite.html ./test/selenium/PublicWebsite_Suite.html ./test/selenium/SystemAdmin_Content_Suite.html ./test/selenium/SystemAdmin_MetaData_Suite.html
killall Xvfb
And I can see the result of the most recent test (you can see the name of my jenkins task folder)
http://<JENKINS.MY.COMPANY>/job/seleniumRegressionTest/ws/seleniumResults/index.html
Earlier tests are all saved on the Jenkins server, so I can view them if I need to.
Allure always set BROKEN status for tests that failed before assertion.
But often tests fail because of existing bug before assertion (like login not working).
Is there a possibility to mark tests that have issue assigned and have failed - as FAILED, not BROKEN in Allure?
The best way is to fix it in Allure adaptor and replace default one with your version.
I have done the same with java adaptor, we were need to add additional execeptions as FAILED and some description to all tests.
I'm doing a bit of reasearch on Jenkins and TestLink and I was wondering - is it possible to automatically generate new project/job in TestLink or does it always have to be done manually? The issue is that I would like to run tests on Jenkins and then all the test that failed should create a new job in TestLink.
Is it possible and how to do it?
With default TestLink Jenkins plugin you can't create new projects automatically on TestLink. You will end up in below type of error
FATAL: Test Project (name:TestSample1234) does not exist.
br.eti.kinoshita.testlinkjavaapi.util.TestLinkAPIException: Test Project (name:TestSample1234) does not exist.
at br.eti.kinoshita.testlinkjavaapi.BaseService.checkResponseError(BaseService.java:113)
at br.eti.kinoshita.testlinkjavaapi.BaseService.executeXmlRpcCall(BaseService.java:91)
at br.eti.kinoshita.testlinkjavaapi.TestProjectService.getTestProjectByName(TestProjectService.java:112)
at br.eti.kinoshita.testlinkjavaapi.TestLinkAPI.getTestProjectByName(TestLinkAPI.java:438)
at hudson.plugins.testlink.TestLinkBuilder.getTestLinkSite(TestLinkBuilder.java:309)
at hudson.plugins.testlink.TestLinkBuilder.perform(TestLinkBuilder.java:186)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782)
at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.build(MavenModuleSetBuild.java:906)
at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:857)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
at hudson.model.Run.execute(Run.java:1738)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:529)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
ERROR: Error communicating with TestLink. Check your TestLink configuration.
Finished: FAILURE
Maybe you can achieve this by customizing the Jenkin plugin, please have look in plugin code [1] and [2]
or asked from testlink forum
[1] https://github.com/jenkinsci/testlink-plugin
[2] https://github.com/kinow/testlink-java-api
Under Test Execution -> Test Reports and Metrics you will be able to filter out only the failed tests.
This can be exported, the icon is quite hidden on the top left, as an excel.
Possibly doing some work on that file you can be able to generate a test suite with only the failed tests
None of the plugins in the reporting section of Maven execute if there is a unit test failure.
I found that I can set maven.test.failure.ignore=true here - http://jira.codehaus.org/browse/SUREFIRE-247 The problem to this approach is now our hudson builds are successful even if there are unit test failures.
What I would really like to do is set the reporting plugin maven-surefire-report-plugin to run with the build plugins on a phase but I can't get this to work.
Any idea on how to get the Maven reporting plugins to execute if a unit test failure occurs?
Firstly run: mvn test OR mvn install. Then, if the tests fail, please run the following target to generate the reports for the test results executed above: mvn -Dmaven.test.skip=true surefire-report:report
In the link you posted:
With the latest version (2.1.2), I get
a message saying that "There are some
test failure," but I get no reports
anywhere whether or not I specify that
variable, or whether or not I specify
"testFailureIgnore" in the plugin
config. I got the reports fine with
2.0, but not with 2.1.2.
Do you need version 2,1 or can you work with a 2.0 version of Maven?
The error you see with 2.1.2 is because of forkmode settings which you need to perform in the plugin.
set forkmode=never and try it (I susppect there might problem in your useSystemclassloader property).
Otherwise make use of maven-surefire plugin version 2.5 which should definitel work and generate surefire rpeorts even though few test fails.
Please make use of surefire-report:report-only plugin if the reports are already generated after execution.
I had the same issue and it is due to a wrong call to the report plugin.
The correct execution of maven command is: mvn surefire-report:report
This will run the test phase by itself and if it fails it will generate the report anyway.
Check the documentation:
http://maven.apache.org/surefire/maven-surefire-report-plugin/report-mojo.html
Hope this helps!! :D