Allure always set BROKEN status for tests that failed before assertion.
But often tests fail because of existing bug before assertion (like login not working).
Is there a possibility to mark tests that have issue assigned and have failed - as FAILED, not BROKEN in Allure?
The best way is to fix it in Allure adaptor and replace default one with your version.
I have done the same with java adaptor, we were need to add additional execeptions as FAILED and some description to all tests.
Related
I have a Bamboo job for mulesoft code, with steps as code checkout, build, test which generates coverage report, deploy.
Test step is failing with message 'Failing task since test cases were expected but none were found.' and coverage report is generated in specific path.
When I remove the specific path, job is success but coverage is not generated.
I tried enabling 'maven return code', which makes the job success, but can't see coverage report.
Give the path **/target/*, enable maven return code - job success, coverage not generated
Give the path **/target/*, disable maven return code - job failed, coverage generated
I found the fix, I had to upgrade munit plugin to higher version ! This fixed the issue !
This should be very late answer but I got same issue with Java + Maven + TestNG + Bamboo and resolved with the respective bamboo task
The below option should be unchecked
The build will produce test results - Uncheck
Maven task configuration
I'm doing a bit of reasearch on Jenkins and TestLink and I was wondering - is it possible to automatically generate new project/job in TestLink or does it always have to be done manually? The issue is that I would like to run tests on Jenkins and then all the test that failed should create a new job in TestLink.
Is it possible and how to do it?
With default TestLink Jenkins plugin you can't create new projects automatically on TestLink. You will end up in below type of error
FATAL: Test Project (name:TestSample1234) does not exist.
br.eti.kinoshita.testlinkjavaapi.util.TestLinkAPIException: Test Project (name:TestSample1234) does not exist.
at br.eti.kinoshita.testlinkjavaapi.BaseService.checkResponseError(BaseService.java:113)
at br.eti.kinoshita.testlinkjavaapi.BaseService.executeXmlRpcCall(BaseService.java:91)
at br.eti.kinoshita.testlinkjavaapi.TestProjectService.getTestProjectByName(TestProjectService.java:112)
at br.eti.kinoshita.testlinkjavaapi.TestLinkAPI.getTestProjectByName(TestLinkAPI.java:438)
at hudson.plugins.testlink.TestLinkBuilder.getTestLinkSite(TestLinkBuilder.java:309)
at hudson.plugins.testlink.TestLinkBuilder.perform(TestLinkBuilder.java:186)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782)
at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.build(MavenModuleSetBuild.java:906)
at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:857)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
at hudson.model.Run.execute(Run.java:1738)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:529)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
ERROR: Error communicating with TestLink. Check your TestLink configuration.
Finished: FAILURE
Maybe you can achieve this by customizing the Jenkin plugin, please have look in plugin code [1] and [2]
or asked from testlink forum
[1] https://github.com/jenkinsci/testlink-plugin
[2] https://github.com/kinow/testlink-java-api
Under Test Execution -> Test Reports and Metrics you will be able to filter out only the failed tests.
This can be exported, the icon is quite hidden on the top left, as an excel.
Possibly doing some work on that file you can be able to generate a test suite with only the failed tests
I’m trying to integrate Bugzilla Testopia with Jenkins with the aid of the Testopia Plugin for Jenkins.
The general configuration is probably fine as the connection between Testopia and Jenkins is well established (Jenkins log says: ‘Connecting to Testopia to retrieve automated test cases’ and no error occurs then). However I’m unable to retrieve any information concerning Test Runs/Test Cases etc. from Testopia.
Moreover I cannot perform any of ‘Iterative Test Build Steps’. If I want to ‘Execute Shell’ in ‘Iterative Test Build Steps’ with Testopia Plugin no operation is carried out (even if I try: echo 12345 etc.). If I use ‘Single Test Build Steps‘ then shell command is executed.
My goal is to retrieve test class name from Testopia (it is stored in Testopia Test Case’s field - Automation/Scripts ) and then run maven build from Jenkins with this class name set as the parameter. Afterwards depending on the Jenkins build success or failure status I’d like to update the Test Case Status in Testopia.
How to fetch any information from Testopia into Jenkins?
Why any of ‘Iterative Test Build Steps’ is not executed?
Any clues? - Testopia Plugin site example wasn't too helpful for me.
Both Bugzilla and Jenkins are hosted on the same Ubuntu 14.04. I've got the latest stable versions of Jenkins, Bugzilla and Testopia.
Thanks in advance,
M.
EDIT:
Well this debugging does not work for me. I added a new log recorder with the 'ALL' level chosen and I cannot see any additional log neither in job's console output nor in the newly created logger output.
Maybe sth is wrong with my Testopia installation? Some more details concerning my configuration:
I've got Testopia installed on the same machine (as Jenkins) and usually I access it through: http://'ip_address'/bugzilla
- in Testopia plugin configuration my URL to Testopia installation is: http://'ip_address'/bugzilla/xmlrpc.cgi
- I've got only one bugzilla account - these admin's credentials I use in Jenkins
- sometimes in Jenkins I can see a warning concerning improper reverse proxy configuraiton - maybe it has sth to do with the problem
After job execution all Testopia's fields are 0 - Run Id, Build Id etc. - which obviously indicates that no information was successfully retrieved from Testopia.
Any ideas how to check why I cannot retrieve any information from Testopia?
EDIT 2:
In the meantime I think I've found a clue in the jenkins.log file in jenkins installation directory:
Exceptions like these occur:
INFO: TESTOPIA_TEST_SUITE_3 #13 main build action completed: SUCCESS
org.apache.xmlrpc.XmlRpcException: The requested method 'TestRun.get' was not found.
at org.apache.xmlrpc.client.XmlRpcStreamTransport.readResponse(XmlRpcStreamTransport.java:197)
...
org.apache.xmlrpc.XmlRpcException: The requested method 'TestRun.get_test_cases' was not found.
at org.apache.xmlrpc.client.XmlRpcStreamTransport.readResponse(XmlRpcStreamTransport.java:197)
Shall I insert the full stacktrace?
It looks like that plugin logs a fair amount of information, though not all of it to the build console output itself.
To debug further, you could try adding a new log recorder for the logger jenkins.plugins.testopia (with log level "all"), run a build, then refresh the web page for the newly-created log recorder to see the output.
You should at least see "Filtering for automated test cases" after connection, information about each test case found, and then log output for each iterative build step as it's run on each test case.
I got the similiar problem when I use python xmlrpc to commnicate with my bugzilla-testopia server.
I chekc the error code and find XMLRPC.pm under my bugzilla install location : "./WebService/Server/XMLRPC.pm".
I know the error was thrown by this sub function, which checks login status and forward to modules where the moduel.function must be in PUBLIC_METHODS:
sub handle_login {
...
if (none { $_ eq $method } $class->PUBLIC_METHODS) {
ThrowCodeError('unknown_method', { method => $full_method });
}
...
}
I don't know why but TestCase.get could not be found in PUBLIC_METHODS, so I just marked them as comment and then it works. You can do this quick way to make sure your client setting is correct. Then you should solve the "PUBLIC_METHODS" problem next.
I have problem in Bamboo 4.4.4:
I've created plan which start some tests and then it generate TestReport.xml.TestReport
I try to parse it with JUnit Parser, but in Test Results Bamboo show only successfull tests and not even one failed.
Any suggestions?
P.S. I atached generated my TestReport.xml and xml-file, that created Bamboo after parse.buildresults19.txt
If your having issues with getting the tests to report correctly please contact Atlassian Support - they can help you sort it out quicker than Stack Overflow!
Thanks
James Dumay
Bamboo Product Manager
None of the plugins in the reporting section of Maven execute if there is a unit test failure.
I found that I can set maven.test.failure.ignore=true here - http://jira.codehaus.org/browse/SUREFIRE-247 The problem to this approach is now our hudson builds are successful even if there are unit test failures.
What I would really like to do is set the reporting plugin maven-surefire-report-plugin to run with the build plugins on a phase but I can't get this to work.
Any idea on how to get the Maven reporting plugins to execute if a unit test failure occurs?
Firstly run: mvn test OR mvn install. Then, if the tests fail, please run the following target to generate the reports for the test results executed above: mvn -Dmaven.test.skip=true surefire-report:report
In the link you posted:
With the latest version (2.1.2), I get
a message saying that "There are some
test failure," but I get no reports
anywhere whether or not I specify that
variable, or whether or not I specify
"testFailureIgnore" in the plugin
config. I got the reports fine with
2.0, but not with 2.1.2.
Do you need version 2,1 or can you work with a 2.0 version of Maven?
The error you see with 2.1.2 is because of forkmode settings which you need to perform in the plugin.
set forkmode=never and try it (I susppect there might problem in your useSystemclassloader property).
Otherwise make use of maven-surefire plugin version 2.5 which should definitel work and generate surefire rpeorts even though few test fails.
Please make use of surefire-report:report-only plugin if the reports are already generated after execution.
I had the same issue and it is due to a wrong call to the report plugin.
The correct execution of maven command is: mvn surefire-report:report
This will run the test phase by itself and if it fails it will generate the report anyway.
Check the documentation:
http://maven.apache.org/surefire/maven-surefire-report-plugin/report-mojo.html
Hope this helps!! :D